2024.11.05 18:29:59.599647 [ 577 ] {} <Warning> Context: Server was built in debug mode. It will work slowly.
2024.11.05 18:29:59.600594 [ 577 ] {} <Warning> Context: ThreadFuzzer is enabled. Application will run slowly and unstable.
2024.11.05 18:29:59.986579 [ 577 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
2024.11.05 18:29:59.994439 [ 577 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.11.05 18:30:12.301533 [ 577 ] {} <Warning> CloudPlacementInfo: Placement info has not been loaded
2024.11.05 18:30:15.942870 [ 577 ] {} <Warning> KeeperLogStore: No logs exists in /var/lib/clickhouse/coordination/logs. It's Ok if it's the first run of clickhouse-keeper.
2024.11.05 18:30:15.958246 [ 577 ] {} <Warning> KeeperLogStore: Removing all changelogs
2024.11.05 18:30:15.967350 [ 577 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.11.05 18:30:15.982019 [ 577 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.11.05 18:30:15.982693 [ 577 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.11.05 18:30:16.066034 [ 1211 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.11.05 18:30:17.381116 [ 577 ] {} <Warning> Access(local_directory): File /var/lib/clickhouse/access/users.list doesn't exist
2024.11.05 18:30:17.389117 [ 577 ] {} <Warning> Access(local_directory): Recovering lists in directory /var/lib/clickhouse/access/
2024.11.05 18:30:21.145768 [ 577 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.11.05 18:31:59.534789 [ 1401 ] {} <Error> ZooKeeperClient: Code: 33. DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4. (CANNOT_READ_ALL_DATA), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<unsigned long&, String>(int, FormatStringHelperImpl<std::type_identity<unsigned long&>::type, std::type_identity<String>::type>, unsigned long&, String&&) @ 0x000000000afd384c
6. /build/src/IO/ReadBuffer.h:199: DB::ReadBuffer::readStrict(char*, unsigned long) @ 0x000000000afd366f
7. /build/src/IO/ReadHelpers.h:118: void DB::readPODBinary<int>(int&, DB::ReadBuffer&) @ 0x00000000133870e2
8. /build/src/IO/ReadHelpers.h:1303: void DB::readBinary<int>(int&, DB::ReadBuffer&) @ 0x000000001338709d
9. /build/src/IO/ReadHelpers.h:1340: void DB::readBinaryEndian<(std::endian)64206, int>(int&, DB::ReadBuffer&) @ 0x000000001d41d53d
10. /build/src/IO/ReadHelpers.h:1353: void DB::readBinaryBigEndian<int>(int&, DB::ReadBuffer&) @ 0x000000001d41d4fd
11. /build/src/Common/ZooKeeper/ZooKeeperIO.h:51: void Coordination::read<int>(int&, DB::ReadBuffer&) @ 0x000000001d417c3d
12. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:289: void Coordination::ZooKeeper::read<int>(int&) @ 0x000000001e252bed
13. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:807: Coordination::ZooKeeper::receiveEvent() @ 0x000000001e247f8c
14. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:773: Coordination::ZooKeeper::receiveThread() @ 0x000000001e2479d9
15. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:398: Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1::operator()() const @ 0x000000001e24de98
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24de75
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e24de59
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&) @ 0x000000001e24ddbd
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&&)::'lambda'()::operator()() @ 0x000000001e24dce5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24dc55
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001e24dc35
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001e24dc15
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001e24dbe0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007fa40dd5dac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 18:31:59.737559 [ 1390 ] {} <Error> ZooKeeperClient: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::1]:52378 -> [::1]:9181). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:169: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e9244
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:303: Coordination::ZooKeeper::flushWriteBuffer() @ 0x000000001e23dffd
9. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:707: Coordination::ZooKeeper::sendThread() @ 0x000000001e246c69
10. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:397: Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0::operator()() const @ 0x000000001e24d658
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24d635
12. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e24d619
13. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0&, std::tuple<>&) @ 0x000000001e24d57d
14. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0&&)::'lambda'()::operator()() @ 0x000000001e24d4a5
15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24d415
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001e24d3f5
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001e24d3d5
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001e24d3a0
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
21. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
22. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
24. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
25. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
26. ? @ 0x00007fa40dd5dac3
27. ? @ 0x00007fa40ddef850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 18:32:00.048453 [ 1401 ] {} <Error> ZooKeeperClient: Poco::Exception. Code: 1000, e.code() = 107, Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/base/poco/Foundation/src/Exception.cpp:145: Poco::RuntimeException::RuntimeException(String const&, int) @ 0x0000000022a0e847
4. /build/base/poco/Foundation/src/Exception.cpp:164: Poco::IOException::IOException(String const&, int) @ 0x0000000022a12987
5. /build/base/poco/Net/src/NetException.cpp:26: Poco::Net::NetException::NetException(String const&, int) @ 0x0000000022899367
6. /build/base/poco/Net/src/SocketImpl.cpp:995: Poco::Net::SocketImpl::error(int, String const&) @ 0x00000000228a8cab
7. /build/base/poco/Net/src/SocketImpl.cpp:914: Poco::Net::SocketImpl::error() @ 0x00000000228a7649
8. /build/base/poco/Net/src/SocketImpl.cpp:272: Poco::Net::SocketImpl::shutdown() @ 0x00000000228aa072
9. /build/base/poco/Net/src/StreamSocket.cpp:109: Poco::Net::StreamSocket::shutdown() @ 0x00000000228add9e
10. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:1032: Coordination::ZooKeeper::finalize(bool, bool, String const&) @ 0x000000001e23ece9
11. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:795: Coordination::ZooKeeper::receiveThread() @ 0x000000001e247ddf
12. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:398: Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1::operator()() const @ 0x000000001e24de98
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24de75
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e24de59
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&) @ 0x000000001e24ddbd
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&&)::'lambda'()::operator()() @ 0x000000001e24dce5
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24dc55
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001e24dc35
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001e24dc15
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001e24dbe0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007fa40dd5dac3
29. ? @ 0x00007fa40ddef850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 18:32:00.799936 [ 580 ] {285d69ce-fbd1-4572-ae75-27b3261f3545} <Warning> InterpreterCreateQuery: ATTACH TABLE query with full table definition is not recommended: use either ATTACH TABLE hits_v1; to attach existing table or CREATE TABLE hits_v1 <table definition>; to create new table or ATTACH TABLE hits_v1 FROM '/path/to/data/' <table definition>; to create new table and attach data.
2024.11.05 18:32:12.424931 [ 580 ] {bce04a25-c5e4-4f88-a630-4d5216db3814} <Warning> InterpreterCreateQuery: ATTACH TABLE query with full table definition is not recommended: use either ATTACH TABLE visits_v1; to attach existing table or CREATE TABLE visits_v1 <table definition>; to create new table or ATTACH TABLE visits_v1 FROM '/path/to/data/' <table definition>; to create new table and attach data.
2024.11.05 18:32:41.267148 [ 1205 ] {} <Error> RaftInstance: failed to accept a rpc connection due to error 125, Operation canceled
2024.11.05 18:32:48.383026 [ 1468 ] {} <Warning> Context: Server was built in debug mode. It will work slowly.
2024.11.05 18:32:48.390248 [ 1468 ] {} <Warning> Context: ThreadFuzzer is enabled. Application will run slowly and unstable.
2024.11.05 18:32:49.387641 [ 1468 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
2024.11.05 18:32:49.396090 [ 1468 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.11.05 18:33:01.485670 [ 1468 ] {} <Warning> CloudPlacementInfo: Placement info has not been loaded
2024.11.05 18:33:05.172940 [ 1468 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.11.05 18:33:05.433383 [ 1468 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.11.05 18:33:05.434149 [ 1468 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.11.05 18:33:05.498920 [ 2109 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.11.05 18:33:21.790259 [ 1468 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.11.05 18:35:17.693336 [ 2199 ] {} <Error> ZooKeeperClient: Code: 33. DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4. (CANNOT_READ_ALL_DATA), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<unsigned long&, String>(int, FormatStringHelperImpl<std::type_identity<unsigned long&>::type, std::type_identity<String>::type>, unsigned long&, String&&) @ 0x000000000afd384c
6. /build/src/IO/ReadBuffer.h:199: DB::ReadBuffer::readStrict(char*, unsigned long) @ 0x000000000afd366f
7. /build/src/IO/ReadHelpers.h:118: void DB::readPODBinary<int>(int&, DB::ReadBuffer&) @ 0x00000000133870e2
8. /build/src/IO/ReadHelpers.h:1303: void DB::readBinary<int>(int&, DB::ReadBuffer&) @ 0x000000001338709d
9. /build/src/IO/ReadHelpers.h:1340: void DB::readBinaryEndian<(std::endian)64206, int>(int&, DB::ReadBuffer&) @ 0x000000001d41d53d
10. /build/src/IO/ReadHelpers.h:1353: void DB::readBinaryBigEndian<int>(int&, DB::ReadBuffer&) @ 0x000000001d41d4fd
11. /build/src/Common/ZooKeeper/ZooKeeperIO.h:51: void Coordination::read<int>(int&, DB::ReadBuffer&) @ 0x000000001d417c3d
12. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:289: void Coordination::ZooKeeper::read<int>(int&) @ 0x000000001e252bed
13. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:807: Coordination::ZooKeeper::receiveEvent() @ 0x000000001e247f8c
14. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:773: Coordination::ZooKeeper::receiveThread() @ 0x000000001e2479d9
15. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:398: Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1::operator()() const @ 0x000000001e24de98
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24de75
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e24de59
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&) @ 0x000000001e24ddbd
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&&)::'lambda'()::operator()() @ 0x000000001e24dce5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24dc55
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001e24dc35
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001e24dc15
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001e24dbe0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f16e3d3bac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 18:35:17.947345 [ 2294 ] {} <Error> ZooKeeperClient: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::1]:56280 -> [::1]:9181). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:169: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e9244
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:303: Coordination::ZooKeeper::flushWriteBuffer() @ 0x000000001e23dffd
9. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:707: Coordination::ZooKeeper::sendThread() @ 0x000000001e246c69
10. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:397: Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0::operator()() const @ 0x000000001e24d658
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24d635
12. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e24d619
13. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0&, std::tuple<>&) @ 0x000000001e24d57d
14. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0&&)::'lambda'()::operator()() @ 0x000000001e24d4a5
15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24d415
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001e24d3f5
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001e24d3d5
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001e24d3a0
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
21. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
22. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
24. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
25. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
26. ? @ 0x00007f16e3d3bac3
27. ? @ 0x00007f16e3dcd850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 18:35:18.121510 [ 2199 ] {} <Error> ZooKeeperClient: Poco::Exception. Code: 1000, e.code() = 107, Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/base/poco/Foundation/src/Exception.cpp:145: Poco::RuntimeException::RuntimeException(String const&, int) @ 0x0000000022a0e847
4. /build/base/poco/Foundation/src/Exception.cpp:164: Poco::IOException::IOException(String const&, int) @ 0x0000000022a12987
5. /build/base/poco/Net/src/NetException.cpp:26: Poco::Net::NetException::NetException(String const&, int) @ 0x0000000022899367
6. /build/base/poco/Net/src/SocketImpl.cpp:995: Poco::Net::SocketImpl::error(int, String const&) @ 0x00000000228a8cab
7. /build/base/poco/Net/src/SocketImpl.cpp:914: Poco::Net::SocketImpl::error() @ 0x00000000228a7649
8. /build/base/poco/Net/src/SocketImpl.cpp:272: Poco::Net::SocketImpl::shutdown() @ 0x00000000228aa072
9. /build/base/poco/Net/src/StreamSocket.cpp:109: Poco::Net::StreamSocket::shutdown() @ 0x00000000228add9e
10. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:1032: Coordination::ZooKeeper::finalize(bool, bool, String const&) @ 0x000000001e23ece9
11. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:795: Coordination::ZooKeeper::receiveThread() @ 0x000000001e247ddf
12. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:398: Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1::operator()() const @ 0x000000001e24de98
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24de75
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e24de59
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&) @ 0x000000001e24ddbd
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&&)::'lambda'()::operator()() @ 0x000000001e24dce5
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24dc55
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001e24dc35
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001e24dc15
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001e24dbe0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f16e3d3bac3
29. ? @ 0x00007f16e3dcd850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 18:43:48.491395 [ 2132 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_71_14}: Code: 241. DB::Exception: Memory tracker (for thread): fault injected (at specific point). (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*, char const*>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type>, char const*&&, char const*&&) @ 0x00000000132eb36c
6. /build/src/Common/MemoryTracker.cpp:184: MemoryTracker::injectFault() const @ 0x00000000132e875f
7. /build/src/Common/CurrentMemoryTracker.cpp:129: CurrentMemoryTracker::injectFault() @ 0x0000000013257305
8. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:149: DB::MergePlainMergeTreeTask::finish() @ 0x000000001d140118
9. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:71: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f876
10. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
11. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
12. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001cbaa415
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001cbaa3dd
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 18:46:21.921609 [ 2139 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_91_18}: Code: 241. DB::Exception: Memory tracker (for thread): fault injected (at specific point). (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*, char const*>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type>, char const*&&, char const*&&) @ 0x00000000132eb36c
6. /build/src/Common/MemoryTracker.cpp:184: MemoryTracker::injectFault() const @ 0x00000000132e875f
7. /build/src/Common/CurrentMemoryTracker.cpp:129: CurrentMemoryTracker::injectFault() @ 0x0000000013257305
8. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:149: DB::MergePlainMergeTreeTask::finish() @ 0x000000001d140118
9. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:71: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f876
10. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
11. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
12. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001cbaa415
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001cbaa3dd
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 18:47:03.967485 [ 2131 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {d830f0a7-c4af-4441-961e-a7e3a729e583::202411_1_101_20}: Code: 241. DB::Exception: Memory tracker (for thread): fault injected (at specific point). (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*, char const*>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type>, char const*&&, char const*&&) @ 0x00000000132eb36c
6. /build/src/Common/MemoryTracker.cpp:184: MemoryTracker::injectFault() const @ 0x00000000132e875f
7. /build/src/Common/CurrentMemoryTracker.cpp:129: CurrentMemoryTracker::injectFault() @ 0x0000000013257305
8. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:149: DB::MergePlainMergeTreeTask::finish() @ 0x000000001d140118
9. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:71: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f876
10. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
11. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
12. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001cbaa415
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001cbaa3dd
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 18:59:06.035652 [ 2129 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_186_39}: Code: 241. DB::Exception: Memory tracker (for thread): fault injected (at specific point). (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*, char const*>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type>, char const*&&, char const*&&) @ 0x00000000132eb36c
6. /build/src/Common/MemoryTracker.cpp:184: MemoryTracker::injectFault() const @ 0x00000000132e875f
7. /build/src/Common/CurrentMemoryTracker.cpp:129: CurrentMemoryTracker::injectFault() @ 0x0000000013257305
8. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:149: DB::MergePlainMergeTreeTask::finish() @ 0x000000001d140118
9. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:71: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f876
10. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
11. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
12. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001cbaa415
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001cbaa3dd
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 18:59:48.863157 [ 2138 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {d830f0a7-c4af-4441-961e-a7e3a729e583::202411_1_201_40}: Code: 241. DB::Exception: Memory tracker (for thread): fault injected (at specific point). (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*, char const*>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type>, char const*&&, char const*&&) @ 0x00000000132eb36c
6. /build/src/Common/MemoryTracker.cpp:184: MemoryTracker::injectFault() const @ 0x00000000132e875f
7. /build/src/Common/CurrentMemoryTracker.cpp:129: CurrentMemoryTracker::injectFault() @ 0x0000000013257305
8. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:149: DB::MergePlainMergeTreeTask::finish() @ 0x000000001d140118
9. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:71: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f876
10. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
11. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
12. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001cbaa415
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001cbaa3dd
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:03:00.490346 [ 2143 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {d830f0a7-c4af-4441-961e-a7e3a729e583::202411_1_226_45}: Code: 241. DB::Exception: Memory tracker (for thread): fault injected (at specific point). (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*, char const*>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type>, char const*&&, char const*&&) @ 0x00000000132eb36c
6. /build/src/Common/MemoryTracker.cpp:184: MemoryTracker::injectFault() const @ 0x00000000132e875f
7. /build/src/Common/CurrentMemoryTracker.cpp:129: CurrentMemoryTracker::injectFault() @ 0x0000000013257305
8. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:149: DB::MergePlainMergeTreeTask::finish() @ 0x000000001d140118
9. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:71: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f876
10. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
11. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
12. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001cbaa415
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001cbaa3dd
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:03:39.953159 [ 2141 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0aef53a4-984b-414b-bb72-6836bf38bb48::202411_1_191_81}: Code: 241. DB::Exception: Memory tracker (for thread): fault injected (at specific point). (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*, char const*>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type>, char const*&&, char const*&&) @ 0x00000000132eb36c
6. /build/src/Common/MemoryTracker.cpp:184: MemoryTracker::injectFault() const @ 0x00000000132e875f
7. /build/src/Common/CurrentMemoryTracker.cpp:129: CurrentMemoryTracker::injectFault() @ 0x0000000013257305
8. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:149: DB::MergePlainMergeTreeTask::finish() @ 0x000000001d140118
9. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:71: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f876
10. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
11. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
12. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001cbaa415
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001cbaa3dd
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:06:00.064912 [ 2133 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {29fdad69-1244-4c25-af3c-70c43e910f15::202411_1_438_193}: Code: 241. DB::Exception: Memory tracker (for thread): fault injected (at specific point). (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*, char const*>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type>, char const*&&, char const*&&) @ 0x00000000132eb36c
6. /build/src/Common/MemoryTracker.cpp:184: MemoryTracker::injectFault() const @ 0x00000000132e875f
7. /build/src/Common/CurrentMemoryTracker.cpp:129: CurrentMemoryTracker::injectFault() @ 0x0000000013257305
8. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:149: DB::MergePlainMergeTreeTask::finish() @ 0x000000001d140118
9. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:71: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f876
10. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
11. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
12. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001cbaa415
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001cbaa3dd
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:18:21.067949 [ 2138 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0a233324-63e4-4351-8154-d23ef4927e18::202411_1_54_13}: Code: 241. DB::Exception: Memory tracker (for thread): fault injected (at specific point). (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*, char const*>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type>, char const*&&, char const*&&) @ 0x00000000132eb36c
6. /build/src/Common/MemoryTracker.cpp:184: MemoryTracker::injectFault() const @ 0x00000000132e875f
7. /build/src/Common/CurrentMemoryTracker.cpp:129: CurrentMemoryTracker::injectFault() @ 0x0000000013257305
8. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:149: DB::MergePlainMergeTreeTask::finish() @ 0x000000001d140118
9. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:71: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f876
10. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
11. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
12. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001cbaa415
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001cbaa3dd
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:26:30.101915 [ 2142 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0aef53a4-984b-414b-bb72-6836bf38bb48::202411_1_364_164}: Code: 241. DB::Exception: Memory tracker (for thread): fault injected (at specific point). (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*, char const*>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type>, char const*&&, char const*&&) @ 0x00000000132eb36c
6. /build/src/Common/MemoryTracker.cpp:184: MemoryTracker::injectFault() const @ 0x00000000132e875f
7. /build/src/Common/CurrentMemoryTracker.cpp:129: CurrentMemoryTracker::injectFault() @ 0x0000000013257305
8. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:149: DB::MergePlainMergeTreeTask::finish() @ 0x000000001d140118
9. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:71: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f876
10. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
11. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
12. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001cbaa415
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001cbaa3dd
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:27:03.103481 [ 2127 ] {} <Warning> CgroupsMemoryUsageObserver: Exceeded soft memory limit (13.77 GiB)
2024.11.05 19:27:48.624820 [ 2127 ] {} <Warning> CgroupsMemoryUsageObserver: Exceeded soft memory limit (13.77 GiB)
2024.11.05 19:29:19.033948 [ 2127 ] {} <Warning> CgroupsMemoryUsageObserver: Exceeded soft memory limit (13.77 GiB)
2024.11.05 19:29:34.348528 [ 2127 ] {} <Warning> CgroupsMemoryUsageObserver: Exceeded hard memory limit (14.54 GiB)
2024.11.05 19:31:19.367576 [ 2127 ] {} <Warning> CgroupsMemoryUsageObserver: Exceeded soft memory limit (13.77 GiB)
2024.11.05 19:31:34.661434 [ 2127 ] {} <Warning> CgroupsMemoryUsageObserver: Exceeded hard memory limit (14.54 GiB)
2024.11.05 19:33:04.670904 [ 2127 ] {} <Warning> CgroupsMemoryUsageObserver: Exceeded soft memory limit (13.77 GiB)
2024.11.05 19:35:06.277091 [ 2137 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0a233324-63e4-4351-8154-d23ef4927e18::202411_1_73_17}: Code: 241. DB::Exception: Memory tracker (for thread): fault injected (at specific point). (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*, char const*>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type>, char const*&&, char const*&&) @ 0x00000000132eb36c
6. /build/src/Common/MemoryTracker.cpp:184: MemoryTracker::injectFault() const @ 0x00000000132e875f
7. /build/src/Common/CurrentMemoryTracker.cpp:129: CurrentMemoryTracker::injectFault() @ 0x0000000013257305
8. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:149: DB::MergePlainMergeTreeTask::finish() @ 0x000000001d140118
9. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:71: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f876
10. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
11. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
12. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001cbaa415
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001cbaa3dd
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:35:15.564566 [ 2130 ] {} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key s3/dsu/qyvcbsvixtakiljnbdbpsrmvuclcj, total size 122843301, count 122843301, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id 3ca0aa81-0661-433c-96a1-4596fd75f0d1, upload has finished false.
2024.11.05 19:35:15.564855 [ 2130 ] {} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key s3/dsu/qyvcbsvixtakiljnbdbpsrmvuclcj, total size 122843301, count 122843301, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id 3ca0aa81-0661-433c-96a1-4596fd75f0d1, upload has finished false
2024.11.05 19:35:15.604683 [ 2130 ] {} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key s3/dsu/qyvcbsvixtakiljnbdbpsrmvuclcj, total size 122843301, count 122843301, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id 3ca0aa81-0661-433c-96a1-4596fd75f0d1, upload has finished false
2024.11.05 19:35:18.412396 [ 2098 ] {} <Error> RaftInstance: failed to accept a rpc connection due to error 125, Operation canceled
2024.11.05 19:35:22.224604 [ 2811 ] {} <Warning> Context: Server was built in debug mode. It will work slowly.
2024.11.05 19:35:22.225605 [ 2811 ] {} <Warning> Context: ThreadFuzzer is enabled. Application will run slowly and unstable.
2024.11.05 19:35:22.545964 [ 2811 ] {} <Warning> Context: Server logging level is set to 'test' and performance is degraded. This cannot be used in production.
2024.11.05 19:35:22.551798 [ 2811 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
2024.11.05 19:35:22.566221 [ 2811 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.11.05 19:35:47.765898 [ 2811 ] {} <Warning> CloudPlacementInfo: Placement info has not been loaded
2024.11.05 19:35:51.585244 [ 2811 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.11.05 19:35:51.602004 [ 2811 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.11.05 19:35:51.602639 [ 2811 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.11.05 19:35:51.664654 [ 3536 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.11.05 19:36:04.330654 [ 2811 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.11.05 19:36:12.810527 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=6, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:37:52.229165 [ 3743 ] {} <Error> ZooKeeperClient: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::1]:9181). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x00000000134e772c
6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x00000000134e68a0
7. /build/src/IO/ReadBuffer.h:70: DB::ReadBuffer::next() @ 0x000000000af5f46a
8. /build/src/IO/ReadBuffer.h:106: DB::CompressedReadBufferBase::readCompressedData(unsigned long&, unsigned long&, bool) @ 0x0000000019dda592
9. /build/src/Compression/CompressedReadBuffer.cpp:12: DB::CompressedReadBuffer::nextImpl() @ 0x0000000019dda083
10. /build/src/IO/ReadBuffer.h:70: DB::ReadBuffer::next() @ 0x000000000af5f46a
11. /build/src/IO/ReadBuffer.h:106: DB::ReadBuffer::read(char*, unsigned long) @ 0x000000000af5f213
12. /build/src/IO/ReadBuffer.h:197: DB::ReadBuffer::readStrict(char*, unsigned long) @ 0x000000000afd35e8
13. /build/src/IO/ReadHelpers.h:118: void DB::readPODBinary<int>(int&, DB::ReadBuffer&) @ 0x00000000133870e2
14. /build/src/IO/ReadHelpers.h:1303: void DB::readBinary<int>(int&, DB::ReadBuffer&) @ 0x000000001338709d
15. /build/src/IO/ReadHelpers.h:1340: void DB::readBinaryEndian<(std::endian)64206, int>(int&, DB::ReadBuffer&) @ 0x000000001d41d53d
16. /build/src/IO/ReadHelpers.h:1353: void DB::readBinaryBigEndian<int>(int&, DB::ReadBuffer&) @ 0x000000001d41d4fd
17. /build/src/Common/ZooKeeper/ZooKeeperIO.h:51: void Coordination::read<int>(int&, DB::ReadBuffer&) @ 0x000000001d417c3d
18. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:289: void Coordination::ZooKeeper::read<int>(int&) @ 0x000000001e252bed
19. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:807: Coordination::ZooKeeper::receiveEvent() @ 0x000000001e247f8c
20. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:773: Coordination::ZooKeeper::receiveThread() @ 0x000000001e2479d9
21. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:398: Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1::operator()() const @ 0x000000001e24de98
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24de75
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e24de59
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&) @ 0x000000001e24ddbd
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&&)::'lambda'()::operator()() @ 0x000000001e24dce5
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24dc55
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001e24dc35
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001e24dc15
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001e24dbe0
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:37:52.550958 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=6, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:37:52.713190 [ 3732 ] {} <Error> ZooKeeperClient: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::1]:40610 -> [::1]:9181). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:169: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e9244
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:303: Coordination::ZooKeeper::flushWriteBuffer() @ 0x000000001e23dffd
9. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:707: Coordination::ZooKeeper::sendThread() @ 0x000000001e246c69
10. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:397: Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0::operator()() const @ 0x000000001e24d658
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24d635
12. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e24d619
13. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0&, std::tuple<>&) @ 0x000000001e24d57d
14. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_0&&)::'lambda'()::operator()() @ 0x000000001e24d4a5
15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24d415
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001e24d3f5
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001e24d3d5
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001e24d3a0
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
21. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
22. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
24. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
25. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
26. ? @ 0x00007f3c91188ac3
27. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:37:52.992230 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=10, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:37:53.640786 [ 3743 ] {} <Error> ZooKeeperClient: Poco::Exception. Code: 1000, e.code() = 107, Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/base/poco/Foundation/src/Exception.cpp:145: Poco::RuntimeException::RuntimeException(String const&, int) @ 0x0000000022a0e847
4. /build/base/poco/Foundation/src/Exception.cpp:164: Poco::IOException::IOException(String const&, int) @ 0x0000000022a12987
5. /build/base/poco/Net/src/NetException.cpp:26: Poco::Net::NetException::NetException(String const&, int) @ 0x0000000022899367
6. /build/base/poco/Net/src/SocketImpl.cpp:995: Poco::Net::SocketImpl::error(int, String const&) @ 0x00000000228a8cab
7. /build/base/poco/Net/src/SocketImpl.cpp:914: Poco::Net::SocketImpl::error() @ 0x00000000228a7649
8. /build/base/poco/Net/src/SocketImpl.cpp:272: Poco::Net::SocketImpl::shutdown() @ 0x00000000228aa072
9. /build/base/poco/Net/src/StreamSocket.cpp:109: Poco::Net::StreamSocket::shutdown() @ 0x00000000228add9e
10. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:1032: Coordination::ZooKeeper::finalize(bool, bool, String const&) @ 0x000000001e23ece9
11. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:795: Coordination::ZooKeeper::receiveThread() @ 0x000000001e247ddf
12. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:398: Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1::operator()() const @ 0x000000001e24de98
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24de75
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e24de59
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::tuple<>&) @ 0x000000001e24ddbd
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>)::$_1&&)::'lambda'()::operator()() @ 0x000000001e24dce5
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001e24dc55
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001e24dc35
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001e24dc15
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001e24dbe0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:37:53.954706 [ 2816 ] {601cd5d9-c9a9-4b3e-bf60-716b6ba9e0af} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=11, jobs=11). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:59976) (in query: SELECT version(), arrayStringConcat(groupArray(value), ' ') FROM system.build_options WHERE name IN ('GIT_HASH', 'GIT_BRANCH') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:37:54.521667 [ 2816 ] {601cd5d9-c9a9-4b3e-bf60-716b6ba9e0af} <Error> DynamicQueryHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=11, jobs=11). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:37:55.709022 [ 3800 ] {e1db05fc-1a95-499b-bdd9-104fdfbd9a06} <Error> executeQuery: Code: 704. DB::Exception: The query result was not cached because the query contains a non-deterministic function. Use setting `query_cache_nondeterministic_function_handling = 'save'` or `= 'ignore'` to cache the query result regardless or to omit caching. (QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60062) (in query: SELECT version(), arrayStringConcat(groupArray(value), ' ') FROM system.build_options WHERE name IN ('GIT_HASH', 'GIT_BRANCH') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Interpreters/executeQuery.cpp:1205: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb3045
7. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
8. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
9. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
10. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
11. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
12. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
13. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
14. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
15. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
16. ? @ 0x00007f3c91188ac3
17. ? @ 0x00007f3c9121a850

2024.11.05 19:37:55.720114 [ 3800 ] {e1db05fc-1a95-499b-bdd9-104fdfbd9a06} <Error> DynamicQueryHandler: Code: 704. DB::Exception: The query result was not cached because the query contains a non-deterministic function. Use setting `query_cache_nondeterministic_function_handling = 'save'` or `= 'ignore'` to cache the query result regardless or to omit caching. (QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Interpreters/executeQuery.cpp:1205: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb3045
7. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
8. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
9. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
10. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
11. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
12. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
13. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
14. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
15. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
16. ? @ 0x00007f3c91188ac3
17. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:37:56.342851 [ 2816 ] {18a2f812-de0c-47b1-aa57-e2917d282596} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=11, jobs=10). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60150) (in query: SELECT version(), arrayStringConcat(groupArray(value), ' ') FROM system.build_options WHERE name IN ('GIT_HASH', 'GIT_BRANCH') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:110: DB::CompletedPipelineExecutor::execute() @ 0x000000001d512ac8
13. /build/src/Interpreters/executeQuery.cpp:1601: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb6c20
14. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
15. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
16. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850

2024.11.05 19:37:56.376843 [ 2816 ] {18a2f812-de0c-47b1-aa57-e2917d282596} <Error> DynamicQueryHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=11, jobs=10). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:110: DB::CompletedPipelineExecutor::execute() @ 0x000000001d512ac8
13. /build/src/Interpreters/executeQuery.cpp:1601: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb6c20
14. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
15. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
16. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:37:56.605611 [ 3820 ] {93a15805-bc49-47ed-b12b-14f5b894a778} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60218) (in query: SELECT version(), arrayStringConcat(groupArray(value), ' ') FROM system.build_options WHERE name IN ('GIT_HASH', 'GIT_BRANCH') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:37:56.652636 [ 3820 ] {93a15805-bc49-47ed-b12b-14f5b894a778} <Error> DynamicQueryHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:37:57.260790 [ 2816 ] {6a31bf89-9a93-41eb-b8eb-52c474cf0c3b} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60226) (in query: SELECT version(), arrayStringConcat(groupArray(value), ' ') FROM system.build_options WHERE name IN ('GIT_HASH', 'GIT_BRANCH') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:110: DB::CompletedPipelineExecutor::execute() @ 0x000000001d512ac8
13. /build/src/Interpreters/executeQuery.cpp:1601: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb6c20
14. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
15. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
16. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850

2024.11.05 19:37:57.283090 [ 2816 ] {6a31bf89-9a93-41eb-b8eb-52c474cf0c3b} <Error> DynamicQueryHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:110: DB::CompletedPipelineExecutor::execute() @ 0x000000001d512ac8
13. /build/src/Interpreters/executeQuery.cpp:1601: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb6c20
14. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
15. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
16. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:37:57.956374 [ 3838 ] {f0ce453b-2f31-4034-9464-d1f1d7b036d0} <Error> executeQuery: Code: 704. DB::Exception: The query result was not cached because the query contains a non-deterministic function. Use setting `query_cache_nondeterministic_function_handling = 'save'` or `= 'ignore'` to cache the query result regardless or to omit caching. (QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60414) (in query: SELECT version(), arrayStringConcat(groupArray(value), ' ') FROM system.build_options WHERE name IN ('GIT_HASH', 'GIT_BRANCH') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Interpreters/executeQuery.cpp:1205: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb3045
7. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
8. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
9. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
10. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
11. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
12. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
13. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
14. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
15. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
16. ? @ 0x00007f3c91188ac3
17. ? @ 0x00007f3c9121a850

2024.11.05 19:37:57.989497 [ 3838 ] {f0ce453b-2f31-4034-9464-d1f1d7b036d0} <Error> DynamicQueryHandler: Code: 704. DB::Exception: The query result was not cached because the query contains a non-deterministic function. Use setting `query_cache_nondeterministic_function_handling = 'save'` or `= 'ignore'` to cache the query result regardless or to omit caching. (QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Interpreters/executeQuery.cpp:1205: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb3045
7. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
8. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
9. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
10. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
11. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
12. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
13. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
14. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
15. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
16. ? @ 0x00007f3c91188ac3
17. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:37:58.394529 [ 3804 ] {093cd9a1-62bc-4b39-ad01-cd517bc08d6d} <Error> executeQuery: Code: 704. DB::Exception: The query result was not cached because the query contains a non-deterministic function. Use setting `query_cache_nondeterministic_function_handling = 'save'` or `= 'ignore'` to cache the query result regardless or to omit caching. (QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60446) (in query: SELECT version(), arrayStringConcat(groupArray(value), ' ') FROM system.build_options WHERE name IN ('GIT_HASH', 'GIT_BRANCH') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Interpreters/executeQuery.cpp:1205: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb3045
7. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
8. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
9. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
10. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
11. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
12. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
13. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
14. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
15. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
16. ? @ 0x00007f3c91188ac3
17. ? @ 0x00007f3c9121a850

2024.11.05 19:37:58.430000 [ 3804 ] {093cd9a1-62bc-4b39-ad01-cd517bc08d6d} <Error> DynamicQueryHandler: Code: 704. DB::Exception: The query result was not cached because the query contains a non-deterministic function. Use setting `query_cache_nondeterministic_function_handling = 'save'` or `= 'ignore'` to cache the query result regardless or to omit caching. (QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Interpreters/executeQuery.cpp:1205: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb3045
7. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
8. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
9. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
10. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
11. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
12. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
13. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
14. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
15. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
16. ? @ 0x00007f3c91188ac3
17. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:37:59.120787 [ 3800 ] {b47ca38e-b9e9-470f-ad03-f6103b28c4ce} <Error> executeQuery: Code: 704. DB::Exception: The query result was not cached because the query contains a non-deterministic function. Use setting `query_cache_nondeterministic_function_handling = 'save'` or `= 'ignore'` to cache the query result regardless or to omit caching. (QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60464) (in query: SELECT version(), arrayStringConcat(groupArray(value), ' ') FROM system.build_options WHERE name IN ('GIT_HASH', 'GIT_BRANCH') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Interpreters/executeQuery.cpp:1205: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb3045
7. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
8. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
9. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
10. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
11. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
12. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
13. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
14. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
15. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
16. ? @ 0x00007f3c91188ac3
17. ? @ 0x00007f3c9121a850

2024.11.05 19:37:59.183422 [ 3800 ] {b47ca38e-b9e9-470f-ad03-f6103b28c4ce} <Error> DynamicQueryHandler: Code: 704. DB::Exception: The query result was not cached because the query contains a non-deterministic function. Use setting `query_cache_nondeterministic_function_handling = 'save'` or `= 'ignore'` to cache the query result regardless or to omit caching. (QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Interpreters/executeQuery.cpp:1205: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb3045
7. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
8. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
9. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
10. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
11. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
12. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
13. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
14. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
15. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
16. ? @ 0x00007f3c91188ac3
17. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:37:59.598788 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=10, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:38:00.187041 [ 3804 ] {e69ee508-de53-4ea6-a592-f8ac73d9c1a0} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=9, jobs=9). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60518) (in query: SELECT version(), arrayStringConcat(groupArray(value), ' ') FROM system.build_options WHERE name IN ('GIT_HASH', 'GIT_BRANCH') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:110: DB::CompletedPipelineExecutor::execute() @ 0x000000001d512ac8
13. /build/src/Interpreters/executeQuery.cpp:1601: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb6c20
14. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
15. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
16. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850

2024.11.05 19:38:00.206323 [ 3804 ] {e69ee508-de53-4ea6-a592-f8ac73d9c1a0} <Error> DynamicQueryHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=9, jobs=9). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:110: DB::CompletedPipelineExecutor::execute() @ 0x000000001d512ac8
13. /build/src/Interpreters/executeQuery.cpp:1601: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb6c20
14. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
15. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
16. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:38:00.448002 [ 3810 ] {8ce24a89-b0da-4c69-8d59-794779e853c8} <Error> executeQuery: Code: 704. DB::Exception: The query result was not cached because the query contains a non-deterministic function. Use setting `query_cache_nondeterministic_function_handling = 'save'` or `= 'ignore'` to cache the query result regardless or to omit caching. (QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60490) (in query: SELECT version(), arrayStringConcat(groupArray(value), ' ') FROM system.build_options WHERE name IN ('GIT_HASH', 'GIT_BRANCH') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Interpreters/executeQuery.cpp:1205: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb3045
7. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
8. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
9. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
10. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
11. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
12. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
13. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
14. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
15. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
16. ? @ 0x00007f3c91188ac3
17. ? @ 0x00007f3c9121a850

2024.11.05 19:38:00.483557 [ 3810 ] {8ce24a89-b0da-4c69-8d59-794779e853c8} <Error> DynamicQueryHandler: Code: 704. DB::Exception: The query result was not cached because the query contains a non-deterministic function. Use setting `query_cache_nondeterministic_function_handling = 'save'` or `= 'ignore'` to cache the query result regardless or to omit caching. (QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Interpreters/executeQuery.cpp:1205: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb3045
7. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
8. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
9. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
10. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
11. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
12. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
13. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
14. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
15. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
16. ? @ 0x00007f3c91188ac3
17. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:38:00.625480 [ 3800 ] {68508196-f7b6-4a26-8aa9-1d37bc393b3b} <Error> executeQuery: Code: 704. DB::Exception: The query result was not cached because the query contains a non-deterministic function. Use setting `query_cache_nondeterministic_function_handling = 'save'` or `= 'ignore'` to cache the query result regardless or to omit caching. (QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60522) (in query: SELECT version(), arrayStringConcat(groupArray(value), ' ') FROM system.build_options WHERE name IN ('GIT_HASH', 'GIT_BRANCH') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Interpreters/executeQuery.cpp:1205: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb3045
7. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
8. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
9. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
10. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
11. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
12. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
13. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
14. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
15. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
16. ? @ 0x00007f3c91188ac3
17. ? @ 0x00007f3c9121a850

2024.11.05 19:38:00.677600 [ 3800 ] {68508196-f7b6-4a26-8aa9-1d37bc393b3b} <Error> DynamicQueryHandler: Code: 704. DB::Exception: The query result was not cached because the query contains a non-deterministic function. Use setting `query_cache_nondeterministic_function_handling = 'save'` or `= 'ignore'` to cache the query result regardless or to omit caching. (QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Interpreters/executeQuery.cpp:1205: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb3045
7. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
8. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
9. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
10. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
11. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
12. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
13. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
14. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
15. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
16. ? @ 0x00007f3c91188ac3
17. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:38:03.807848 [ 3828 ] {d6987c46-580f-400b-943c-a9f6608bbeed} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=4, jobs=4). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:43196) (comment: 00068_subquery_in_prewhere.sql) (in query: SELECT count() FROM test.hits PREWHERE UserID IN (SELECT UserID FROM test.hits WHERE CounterID = 800784);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:38:03.833155 [ 3828 ] {d6987c46-580f-400b-943c-a9f6608bbeed} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=4, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:38:08.298302 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=10, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:38:11.699709 [ 3828 ] {d034525f-26a9-49ff-b20a-ddc8ffb30746} <Error> executeQuery: Code: 59. DB::Exception: Illegal type Nullable(Int128) of column __table1.x for filter. Must be UInt8 or Nullable(UInt8). (ILLEGAL_TYPE_OF_COLUMN_FOR_FILTER) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:43222) (comment: 02794_pushdown_invalid_get.sql) (in query: SELECT * FROM (SELECT toInt128(NULL) AS x UNION ALL SELECT materialize(toInt128(-2))) WHERE x;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, String const&>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String const&>::type>, String&&, String const&) @ 0x000000000b499e8c
6. /build/src/Processors/Transforms/FilterTransform.cpp:192: DB::FilterTransform::transformHeader(DB::Block, DB::ActionsDAG const*, String const&, bool) @ 0x000000001da458fb
7. /build/src/Processors/QueryPlan/FilterStep.cpp:43: DB::FilterStep::FilterStep(DB::DataStream const&, std::shared_ptr<DB::ActionsDAG> const&, String, bool) @ 0x000000001dd05bc9
8. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::FilterStep>::__unique_single std::make_unique[abi:v15000]<DB::FilterStep, DB::DataStream const&, std::shared_ptr<DB::ActionsDAG> const&, String const&, bool const&>(DB::DataStream const&, std::shared_ptr<DB::ActionsDAG> const&, String const&, bool const&) @ 0x000000001b5258cd
9. /build/src/Planner/Planner.cpp:347: DB::(anonymous namespace)::addFilterStep(DB::QueryPlan&, DB::FilterAnalysisResult const&, String const&, std::vector<std::shared_ptr<DB::ActionsDAG>, std::allocator<std::shared_ptr<DB::ActionsDAG>>>&) @ 0x000000001b513a48
10. /build/src/Planner/Planner.cpp:1533: DB::Planner::buildPlanForQueryNode() @ 0x000000001b510d70
11. /build/src/Planner/Planner.cpp:0: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000001b50be55
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:234: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000001b50905d
13. /build/src/Interpreters/executeQuery.cpp:1146: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2812
14. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
15. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
16. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850

2024.11.05 19:38:11.703978 [ 3828 ] {d034525f-26a9-49ff-b20a-ddc8ffb30746} <Error> TCPHandler: Code: 59. DB::Exception: Illegal type Nullable(Int128) of column __table1.x for filter. Must be UInt8 or Nullable(UInt8). (ILLEGAL_TYPE_OF_COLUMN_FOR_FILTER), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, String const&>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String const&>::type>, String&&, String const&) @ 0x000000000b499e8c
6. /build/src/Processors/Transforms/FilterTransform.cpp:192: DB::FilterTransform::transformHeader(DB::Block, DB::ActionsDAG const*, String const&, bool) @ 0x000000001da458fb
7. /build/src/Processors/QueryPlan/FilterStep.cpp:43: DB::FilterStep::FilterStep(DB::DataStream const&, std::shared_ptr<DB::ActionsDAG> const&, String, bool) @ 0x000000001dd05bc9
8. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::FilterStep>::__unique_single std::make_unique[abi:v15000]<DB::FilterStep, DB::DataStream const&, std::shared_ptr<DB::ActionsDAG> const&, String const&, bool const&>(DB::DataStream const&, std::shared_ptr<DB::ActionsDAG> const&, String const&, bool const&) @ 0x000000001b5258cd
9. /build/src/Planner/Planner.cpp:347: DB::(anonymous namespace)::addFilterStep(DB::QueryPlan&, DB::FilterAnalysisResult const&, String const&, std::vector<std::shared_ptr<DB::ActionsDAG>, std::allocator<std::shared_ptr<DB::ActionsDAG>>>&) @ 0x000000001b513a48
10. /build/src/Planner/Planner.cpp:1533: DB::Planner::buildPlanForQueryNode() @ 0x000000001b510d70
11. /build/src/Planner/Planner.cpp:0: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000001b50be55
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:234: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000001b50905d
13. /build/src/Interpreters/executeQuery.cpp:1146: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2812
14. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
15. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
16. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850

2024.11.05 19:38:12.073879 [ 3820 ] {5b456046-1ce3-4e3f-8f1e-73736b64f17a} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=21, jobs=21). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:43210) (comment: 00021_1_select_with_in.sql) (in query: select sum(Sign) from test.visits where CounterID in (942285);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:38:12.159473 [ 3820 ] {5b456046-1ce3-4e3f-8f1e-73736b64f17a} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=21, jobs=21). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:38:13.201346 [ 3839 ] {d1ac98fa-0a26-4e4c-8feb-bccdb3b9362a} <Error> executeQuery: Code: 349. DB::Exception: Cannot convert NULL value to non-Nullable type: In scope SELECT CAST(CAST(NULL, 'LowCardinality(Nullable(String))'), 'String'). (CANNOT_INSERT_NULL_IN_ORDINARY_COLUMN) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:43228) (comment: 00974_low_cardinality_cast.sql) (in query: SELECT CAST(CAST(Null AS LowCardinality(Nullable(String))) AS String);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Functions/FunctionsConversion.cpp:4450: DB::(anonymous namespace)::FunctionCast::prepareUnpackDictionaries(std::shared_ptr<DB::IDataType const> const&, std::shared_ptr<DB::IDataType const> const&) const::'lambda0'(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>&, std::shared_ptr<DB::IDataType const> const&, DB::ColumnNullable const*, unsigned long)::operator()(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>&, std::shared_ptr<DB::IDataType const> const&, DB::ColumnNullable const*, unsigned long) const @ 0x000000000adc9598
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000adc9246
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000adc91e0
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000adc9180
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000adc912b
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af4dda7
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af4c5c0
13. /build/src/Functions/FunctionsConversion.cpp:3048: DB::(anonymous namespace)::ExecutableFunctionCast::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000aa6a94f
14. /build/src/Functions/IFunction.h:61: DB::IExecutableFunction::executeDryRunImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000a945a59
15. /build/src/Functions/IFunction.cpp:277: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062f06
16. /build/src/Functions/IFunction.cpp:191: DB::IExecutableFunction::defaultImplementationForConstantArguments(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062a83
17. /build/src/Functions/IFunction.cpp:269: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062de3
18. /build/src/Functions/IFunction.cpp:340: DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000001906408e
19. /build/src/Functions/IFunction.cpp:411: DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000190650b5
20. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6185: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be389
21. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
22. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221
23. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6901: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0ab9c7
24. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8173: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d543
25. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
26. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
27. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
28. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
29. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
30. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
31. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c

2024.11.05 19:38:13.213136 [ 3839 ] {d1ac98fa-0a26-4e4c-8feb-bccdb3b9362a} <Error> TCPHandler: Code: 349. DB::Exception: Cannot convert NULL value to non-Nullable type: In scope SELECT CAST(CAST(NULL, 'LowCardinality(Nullable(String))'), 'String'). (CANNOT_INSERT_NULL_IN_ORDINARY_COLUMN), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Functions/FunctionsConversion.cpp:4450: DB::(anonymous namespace)::FunctionCast::prepareUnpackDictionaries(std::shared_ptr<DB::IDataType const> const&, std::shared_ptr<DB::IDataType const> const&) const::'lambda0'(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>&, std::shared_ptr<DB::IDataType const> const&, DB::ColumnNullable const*, unsigned long)::operator()(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>&, std::shared_ptr<DB::IDataType const> const&, DB::ColumnNullable const*, unsigned long) const @ 0x000000000adc9598
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000adc9246
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000adc91e0
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000adc9180
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000adc912b
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af4dda7
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af4c5c0
13. /build/src/Functions/FunctionsConversion.cpp:3048: DB::(anonymous namespace)::ExecutableFunctionCast::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000aa6a94f
14. /build/src/Functions/IFunction.h:61: DB::IExecutableFunction::executeDryRunImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000a945a59
15. /build/src/Functions/IFunction.cpp:277: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062f06
16. /build/src/Functions/IFunction.cpp:191: DB::IExecutableFunction::defaultImplementationForConstantArguments(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062a83
17. /build/src/Functions/IFunction.cpp:269: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062de3
18. /build/src/Functions/IFunction.cpp:340: DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000001906408e
19. /build/src/Functions/IFunction.cpp:411: DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000190650b5
20. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6185: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be389
21. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
22. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221
23. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6901: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0ab9c7
24. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8173: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d543
25. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
26. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
27. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
28. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
29. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
30. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
31. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c

2024.11.05 19:38:13.819824 [ 3839 ] {d46f2ded-267a-47d2-9b84-5fd1b807fd11} <Error> executeQuery: Code: 349. DB::Exception: Cannot convert NULL value to non-Nullable type: In scope SELECT CAST(CAST(NULL, 'Nullable(String)'), 'String'). (CANNOT_INSERT_NULL_IN_ORDINARY_COLUMN) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:43228) (comment: 00974_low_cardinality_cast.sql) (in query: SELECT CAST(CAST(Null AS Nullable(String)) AS String);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Functions/FunctionsConversion.cpp:4553: DB::(anonymous namespace)::FunctionCast::prepareRemoveNullable(std::shared_ptr<DB::IDataType const> const&, std::shared_ptr<DB::IDataType const> const&, bool) const::'lambda0'(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>&, std::shared_ptr<DB::IDataType const> const&, DB::ColumnNullable const*, unsigned long)::operator()(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>&, std::shared_ptr<DB::IDataType const> const&, DB::ColumnNullable const*, unsigned long) const @ 0x000000000adc8ba0
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000adc89e6
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000adc8980
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000adc8920
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000adc88cb
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af4dda7
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af4c5c0
13. /build/src/Functions/FunctionsConversion.cpp:3048: DB::(anonymous namespace)::ExecutableFunctionCast::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000aa6a94f
14. /build/src/Functions/IFunction.h:61: DB::IExecutableFunction::executeDryRunImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000a945a59
15. /build/src/Functions/IFunction.cpp:277: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062f06
16. /build/src/Functions/IFunction.cpp:191: DB::IExecutableFunction::defaultImplementationForConstantArguments(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062a83
17. /build/src/Functions/IFunction.cpp:269: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062de3
18. /build/src/Functions/IFunction.cpp:340: DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000001906408e
19. /build/src/Functions/IFunction.cpp:411: DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000190650b5
20. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6185: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be389
21. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
22. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221
23. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6901: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0ab9c7
24. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8173: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d543
25. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
26. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
27. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
28. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
29. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
30. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
31. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c

2024.11.05 19:38:13.851753 [ 3839 ] {d46f2ded-267a-47d2-9b84-5fd1b807fd11} <Error> TCPHandler: Code: 349. DB::Exception: Cannot convert NULL value to non-Nullable type: In scope SELECT CAST(CAST(NULL, 'Nullable(String)'), 'String'). (CANNOT_INSERT_NULL_IN_ORDINARY_COLUMN), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Functions/FunctionsConversion.cpp:4553: DB::(anonymous namespace)::FunctionCast::prepareRemoveNullable(std::shared_ptr<DB::IDataType const> const&, std::shared_ptr<DB::IDataType const> const&, bool) const::'lambda0'(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>&, std::shared_ptr<DB::IDataType const> const&, DB::ColumnNullable const*, unsigned long)::operator()(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>&, std::shared_ptr<DB::IDataType const> const&, DB::ColumnNullable const*, unsigned long) const @ 0x000000000adc8ba0
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000adc89e6
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000adc8980
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000adc8920
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000adc88cb
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af4dda7
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af4c5c0
13. /build/src/Functions/FunctionsConversion.cpp:3048: DB::(anonymous namespace)::ExecutableFunctionCast::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000aa6a94f
14. /build/src/Functions/IFunction.h:61: DB::IExecutableFunction::executeDryRunImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000a945a59
15. /build/src/Functions/IFunction.cpp:277: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062f06
16. /build/src/Functions/IFunction.cpp:191: DB::IExecutableFunction::defaultImplementationForConstantArguments(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062a83
17. /build/src/Functions/IFunction.cpp:269: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062de3
18. /build/src/Functions/IFunction.cpp:340: DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000001906408e
19. /build/src/Functions/IFunction.cpp:411: DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000190650b5
20. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6185: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be389
21. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
22. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221
23. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6901: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0ab9c7
24. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8173: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d543
25. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
26. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
27. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
28. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
29. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
30. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
31. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c

2024.11.05 19:38:19.546135 [ 3810 ] {c7800c66-3a81-449a-bc14-11e11b5e853d} <Error> executeQuery: Code: 36. DB::Exception: Expression in arrayJoin cannot depend on lambda argument: x . (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52558) (comment: 02369_analyzer_array_join_function.sql) (in query: SELECT arrayMap(x -> arrayJoin(x), [[1, 2, 3]]);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Planner/PlannerActionsVisitor.cpp:1007: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitFunction(std::shared_ptr<DB::IQueryTreeNode> const&) @ 0x000000001a4a8de0
7. /build/src/Planner/PlannerActionsVisitor.cpp:658: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitImpl(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001a4a7ae9
8. /build/src/Planner/PlannerActionsVisitor.cpp:782: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitLambda(std::shared_ptr<DB::IQueryTreeNode> const&) @ 0x000000001a4ab6ce
9. /build/src/Planner/PlannerActionsVisitor.cpp:986: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitFunction(std::shared_ptr<DB::IQueryTreeNode> const&) @ 0x000000001a4a8a30
10. /build/src/Planner/PlannerActionsVisitor.cpp:658: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitImpl(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001a4a7ae9
11. /build/src/Planner/PlannerActionsVisitor.cpp:636: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visit(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001a4a3508
12. /build/src/Planner/PlannerActionsVisitor.cpp:1043: DB::PlannerActionsVisitor::visit(std::shared_ptr<DB::ActionsDAG>, std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001a4a3218
13. /build/src/Planner/Utils.cpp:242: DB::buildActionsDAGFromExpressionNode(std::shared_ptr<DB::IQueryTreeNode> const&, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::PlannerContext> const&) @ 0x000000001b532aa5
14. /build/src/Planner/PlannerExpressionAnalysis.cpp:360: DB::(anonymous namespace)::analyzeProjection(DB::QueryNode const&, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::PlannerContext> const&, DB::ActionsChain&) @ 0x000000001b5d0429
15. /build/src/Planner/PlannerExpressionAnalysis.cpp:559: DB::buildExpressionAnalysisResult(std::shared_ptr<DB::IQueryTreeNode> const&, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::PlannerContext> const&, DB::PlannerQueryProcessingInfo const&) @ 0x000000001b5ccb7b
16. /build/src/Planner/Planner.cpp:1497: DB::Planner::buildPlanForQueryNode() @ 0x000000001b5109b9
17. /build/src/Planner/Planner.cpp:0: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000001b50be55
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:234: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000001b50905d
19. /build/src/Interpreters/executeQuery.cpp:1146: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2812
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:38:19.550182 [ 3810 ] {c7800c66-3a81-449a-bc14-11e11b5e853d} <Error> TCPHandler: Code: 36. DB::Exception: Expression in arrayJoin cannot depend on lambda argument: x . (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Planner/PlannerActionsVisitor.cpp:1007: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitFunction(std::shared_ptr<DB::IQueryTreeNode> const&) @ 0x000000001a4a8de0
7. /build/src/Planner/PlannerActionsVisitor.cpp:658: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitImpl(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001a4a7ae9
8. /build/src/Planner/PlannerActionsVisitor.cpp:782: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitLambda(std::shared_ptr<DB::IQueryTreeNode> const&) @ 0x000000001a4ab6ce
9. /build/src/Planner/PlannerActionsVisitor.cpp:986: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitFunction(std::shared_ptr<DB::IQueryTreeNode> const&) @ 0x000000001a4a8a30
10. /build/src/Planner/PlannerActionsVisitor.cpp:658: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitImpl(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001a4a7ae9
11. /build/src/Planner/PlannerActionsVisitor.cpp:636: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visit(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001a4a3508
12. /build/src/Planner/PlannerActionsVisitor.cpp:1043: DB::PlannerActionsVisitor::visit(std::shared_ptr<DB::ActionsDAG>, std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001a4a3218
13. /build/src/Planner/Utils.cpp:242: DB::buildActionsDAGFromExpressionNode(std::shared_ptr<DB::IQueryTreeNode> const&, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::PlannerContext> const&) @ 0x000000001b532aa5
14. /build/src/Planner/PlannerExpressionAnalysis.cpp:360: DB::(anonymous namespace)::analyzeProjection(DB::QueryNode const&, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::PlannerContext> const&, DB::ActionsChain&) @ 0x000000001b5d0429
15. /build/src/Planner/PlannerExpressionAnalysis.cpp:559: DB::buildExpressionAnalysisResult(std::shared_ptr<DB::IQueryTreeNode> const&, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::PlannerContext> const&, DB::PlannerQueryProcessingInfo const&) @ 0x000000001b5ccb7b
16. /build/src/Planner/Planner.cpp:1497: DB::Planner::buildPlanForQueryNode() @ 0x000000001b5109b9
17. /build/src/Planner/Planner.cpp:0: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000001b50be55
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:234: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000001b50905d
19. /build/src/Interpreters/executeQuery.cpp:1146: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2812
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:38:19.624027 [ 3810 ] {89a52e8a-2bf9-42ab-b5f2-c021754ba48e} <Error> executeQuery: Code: 36. DB::Exception: Expression in arrayJoin cannot depend on lambda argument: CAST(x, 'Array(UInt8)'_String) . (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52558) (comment: 02369_analyzer_array_join_function.sql) (in query: SELECT arrayMap(x -> arrayJoin(cast(x, 'Array(UInt8)')), [[1, 2, 3]]);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Planner/PlannerActionsVisitor.cpp:1007: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitFunction(std::shared_ptr<DB::IQueryTreeNode> const&) @ 0x000000001a4a8de0
7. /build/src/Planner/PlannerActionsVisitor.cpp:658: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitImpl(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001a4a7ae9
8. /build/src/Planner/PlannerActionsVisitor.cpp:782: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitLambda(std::shared_ptr<DB::IQueryTreeNode> const&) @ 0x000000001a4ab6ce
9. /build/src/Planner/PlannerActionsVisitor.cpp:986: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitFunction(std::shared_ptr<DB::IQueryTreeNode> const&) @ 0x000000001a4a8a30
10. /build/src/Planner/PlannerActionsVisitor.cpp:658: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitImpl(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001a4a7ae9
11. /build/src/Planner/PlannerActionsVisitor.cpp:636: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visit(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001a4a3508
12. /build/src/Planner/PlannerActionsVisitor.cpp:1043: DB::PlannerActionsVisitor::visit(std::shared_ptr<DB::ActionsDAG>, std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001a4a3218
13. /build/src/Planner/Utils.cpp:242: DB::buildActionsDAGFromExpressionNode(std::shared_ptr<DB::IQueryTreeNode> const&, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::PlannerContext> const&) @ 0x000000001b532aa5
14. /build/src/Planner/PlannerExpressionAnalysis.cpp:360: DB::(anonymous namespace)::analyzeProjection(DB::QueryNode const&, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::PlannerContext> const&, DB::ActionsChain&) @ 0x000000001b5d0429
15. /build/src/Planner/PlannerExpressionAnalysis.cpp:559: DB::buildExpressionAnalysisResult(std::shared_ptr<DB::IQueryTreeNode> const&, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::PlannerContext> const&, DB::PlannerQueryProcessingInfo const&) @ 0x000000001b5ccb7b
16. /build/src/Planner/Planner.cpp:1497: DB::Planner::buildPlanForQueryNode() @ 0x000000001b5109b9
17. /build/src/Planner/Planner.cpp:0: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000001b50be55
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:234: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000001b50905d
19. /build/src/Interpreters/executeQuery.cpp:1146: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2812
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:38:19.628189 [ 3810 ] {89a52e8a-2bf9-42ab-b5f2-c021754ba48e} <Error> TCPHandler: Code: 36. DB::Exception: Expression in arrayJoin cannot depend on lambda argument: CAST(x, 'Array(UInt8)'_String) . (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Planner/PlannerActionsVisitor.cpp:1007: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitFunction(std::shared_ptr<DB::IQueryTreeNode> const&) @ 0x000000001a4a8de0
7. /build/src/Planner/PlannerActionsVisitor.cpp:658: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitImpl(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001a4a7ae9
8. /build/src/Planner/PlannerActionsVisitor.cpp:782: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitLambda(std::shared_ptr<DB::IQueryTreeNode> const&) @ 0x000000001a4ab6ce
9. /build/src/Planner/PlannerActionsVisitor.cpp:986: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitFunction(std::shared_ptr<DB::IQueryTreeNode> const&) @ 0x000000001a4a8a30
10. /build/src/Planner/PlannerActionsVisitor.cpp:658: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visitImpl(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001a4a7ae9
11. /build/src/Planner/PlannerActionsVisitor.cpp:636: DB::(anonymous namespace)::PlannerActionsVisitorImpl::visit(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001a4a3508
12. /build/src/Planner/PlannerActionsVisitor.cpp:1043: DB::PlannerActionsVisitor::visit(std::shared_ptr<DB::ActionsDAG>, std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001a4a3218
13. /build/src/Planner/Utils.cpp:242: DB::buildActionsDAGFromExpressionNode(std::shared_ptr<DB::IQueryTreeNode> const&, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::PlannerContext> const&) @ 0x000000001b532aa5
14. /build/src/Planner/PlannerExpressionAnalysis.cpp:360: DB::(anonymous namespace)::analyzeProjection(DB::QueryNode const&, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::PlannerContext> const&, DB::ActionsChain&) @ 0x000000001b5d0429
15. /build/src/Planner/PlannerExpressionAnalysis.cpp:559: DB::buildExpressionAnalysisResult(std::shared_ptr<DB::IQueryTreeNode> const&, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::PlannerContext> const&, DB::PlannerQueryProcessingInfo const&) @ 0x000000001b5ccb7b
16. /build/src/Planner/Planner.cpp:1497: DB::Planner::buildPlanForQueryNode() @ 0x000000001b5109b9
17. /build/src/Planner/Planner.cpp:0: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000001b50be55
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:234: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000001b50905d
19. /build/src/Interpreters/executeQuery.cpp:1146: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2812
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:38:20.193245 [ 3839 ] {ca1103c7-3852-4a4b-981d-9806cde75c95} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=18, jobs=18). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52580) (comment: 02013_bloom_filter_hasAll.sql) (in query: -- verify the expression in WHERE works on non-index col the same way as on index cols
 SELECT count() FROM bftest WHERE hasAll(y, [NULL,-42]) FORMAT Null;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:38:20.239521 [ 3839 ] {ca1103c7-3852-4a4b-981d-9806cde75c95} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=18, jobs=18). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:38:20.803182 [ 3810 ] {cec49054-885f-4db9-9140-e3a094fd1266} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=33, jobs=33). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52558) (comment: 02369_analyzer_array_join_function.sql) (in query: SELECT id, arrayJoin(value_1) FROM test_table;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:38:20.845725 [ 3810 ] {cec49054-885f-4db9-9140-e3a094fd1266} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=33, jobs=33). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:38:23.355393 [ 3838 ] {732ab209-eae1-4d54-845d-d9e5e5fdfe23} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=73): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:43198) (comment: 00065_loyalty_with_storage_join.sql) (in query: INSERT INTO join SELECT UserID, toInt8(if((sum(SearchEngineID = 2) AS yandex) > (sum(SearchEngineID = 3) AS google), yandex / (yandex + google), -google / (yandex + google)) * 10) AS loyalty FROM hits WHERE (SearchEngineID = 2) OR (SearchEngineID = 3) GROUP BY UserID HAVING (yandex + google) > 10;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:38:23.355513 [ 3820 ] {f1e83567-b5b2-43f0-95f7-7b9e3e45810b} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=54): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52570) (comment: 00077_log_tinylog_stripelog.sql) (in query: INSERT INTO test.hits_log SELECT CounterID, AdvEngineID, RegionID, SearchPhrase, UserID FROM test.hits;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:38:24.385422 [ 3838 ] {732ab209-eae1-4d54-845d-d9e5e5fdfe23} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=73): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:38:25.296295 [ 3820 ] {f1e83567-b5b2-43f0-95f7-7b9e3e45810b} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=54): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:38:25.794134 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=13). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:38:27.078594 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:38:33.356561 [ 3800 ] {c770e6f1-935c-47fb-899f-d3d5479db33f} <Error> executeQuery: Code: 36. DB::Exception: Table TTL is not allowed for MergeTree in old syntax. (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:33502) (comment: 01713_table_ttl_old_syntax_zookeeper.sql) (in query: CREATE TABLE ttl_table ( date Date, value UInt64 ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{database}/test_01713_table_ttl', '1', date, date, 8192) TTL date + INTERVAL 2 MONTH;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/MergeTree/registerStorageMergeTree.cpp:705: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000001d0e52ee
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000193086fd
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000193086bd
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001930867d
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019308645
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
13. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
14. /build/src/Interpreters/InterpreterCreateQuery.cpp:1493: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x000000001b01ec0c
15. /build/src/Interpreters/InterpreterCreateQuery.cpp:1313: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b018b66
16. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
17. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
18. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
19. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
20. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
21. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
22. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
23. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
24. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
25. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
26. ? @ 0x00007f3c91188ac3
27. ? @ 0x00007f3c9121a850

2024.11.05 19:38:33.394990 [ 3800 ] {c770e6f1-935c-47fb-899f-d3d5479db33f} <Error> TCPHandler: Code: 36. DB::Exception: Table TTL is not allowed for MergeTree in old syntax. (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/MergeTree/registerStorageMergeTree.cpp:705: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000001d0e52ee
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000193086fd
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000193086bd
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001930867d
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019308645
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
13. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
14. /build/src/Interpreters/InterpreterCreateQuery.cpp:1493: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x000000001b01ec0c
15. /build/src/Interpreters/InterpreterCreateQuery.cpp:1313: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b018b66
16. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
17. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
18. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
19. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
20. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
21. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
22. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
23. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
24. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
25. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
26. ? @ 0x00007f3c91188ac3
27. ? @ 0x00007f3c9121a850

2024.11.05 19:38:33.971577 [ 3800 ] {9aa110d6-3c19-404c-98c6-5ff1676d0c12} <Error> executeQuery: Code: 42. DB::Exception: With extended storage definition syntax storage ReplicatedMergeTree requires 0 to 2 parameters: 
[path in ZooKeeper],
[replica name]

Syntax for the MergeTree table engine:

CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster]
(
    name1 [type1] [DEFAULT|MATERIALIZED|ALIAS expr1] [TTL expr1],
    name2 [type2] [DEFAULT|MATERIALIZED|ALIAS expr2] [TTL expr2],
    ...
    INDEX index_name1 expr1 TYPE type1(...) [GRANULARITY value1],
    INDEX index_name2 expr2 TYPE type2(...) [GRANULARITY value2]
) ENGINE = MergeTree()
ORDER BY expr
[PARTITION BY expr]
[PRIMARY KEY expr]
[SAMPLE BY expr]
[TTL expr [DELETE|TO DISK 'xxx'|TO VOLUME 'xxx'], ...]
[SETTINGS name=value, ...]
[COMMENT 'comment']

See details in documentation: https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/. Other engines of the family support different syntax, see details in the corresponding documentation topics.

If you use the Replicated version of engines, see https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication/.
. (NUMBER_OF_ARGUMENTS_DOESNT_MATCH) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:33502) (comment: 01713_table_ttl_old_syntax_zookeeper.sql) (in query: CREATE TABLE ttl_table (`date` Date, `value` UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{database}/test_01713_table_ttl', '1', date, date, 8192) PARTITION BY date SETTINGS index_granularity = 4125, min_bytes_for_wide_part = 335320873, ratio_of_defaults_for_sparse_serialization = 0., replace_long_file_name_to_hash = false, max_file_name_length = 128, min_bytes_for_full_part_storage = 536870912, compact_parts_max_bytes_to_buffer = 280561274, compact_parts_max_granules_to_buffer = 171, compact_parts_merge_max_bytes_to_prefetch_part = 6523271, merge_max_block_size = 4962, old_parts_lifetime = 337., prefer_fetch_merged_part_size_threshold = 2974024565, vertical_merge_algorithm_min_rows_to_activate = 1000000, vertical_merge_algorithm_min_columns_to_activate = 100, min_merge_bytes_to_use_direct_io = 10737418240, index_granularity_bytes = 10568046, concurrent_part_removal_threshold = 0, allow_vertical_merges_from_compact_to_wide_parts = false, cache_populated_by_fetch = false, marks_compress_block_size = 61920, primary_key_compress_block_size = 31615), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String&, char const* const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String&>::type, std::type_identity<char const* const&>::type>, String const&, String&, char const* const&) @ 0x000000001d0e7068
6. /build/src/Storages/MergeTree/registerStorageMergeTree.cpp:260: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000001d0e0d4b
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000193086fd
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000193086bd
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001930867d
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019308645
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
13. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
14. /build/src/Interpreters/InterpreterCreateQuery.cpp:1493: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x000000001b01ec0c
15. /build/src/Interpreters/InterpreterCreateQuery.cpp:1313: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b018b66
16. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
17. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
18. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
19. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
20. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
21. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
22. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
23. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
24. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
25. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
26. ? @ 0x00007f3c91188ac3
27. ? @ 0x00007f3c9121a850

2024.11.05 19:38:33.992411 [ 3800 ] {9aa110d6-3c19-404c-98c6-5ff1676d0c12} <Error> TCPHandler: Code: 42. DB::Exception: With extended storage definition syntax storage ReplicatedMergeTree requires 0 to 2 parameters: 
[path in ZooKeeper],
[replica name]

Syntax for the MergeTree table engine:

CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster]
(
    name1 [type1] [DEFAULT|MATERIALIZED|ALIAS expr1] [TTL expr1],
    name2 [type2] [DEFAULT|MATERIALIZED|ALIAS expr2] [TTL expr2],
    ...
    INDEX index_name1 expr1 TYPE type1(...) [GRANULARITY value1],
    INDEX index_name2 expr2 TYPE type2(...) [GRANULARITY value2]
) ENGINE = MergeTree()
ORDER BY expr
[PARTITION BY expr]
[PRIMARY KEY expr]
[SAMPLE BY expr]
[TTL expr [DELETE|TO DISK 'xxx'|TO VOLUME 'xxx'], ...]
[SETTINGS name=value, ...]
[COMMENT 'comment']

See details in documentation: https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/. Other engines of the family support different syntax, see details in the corresponding documentation topics.

If you use the Replicated version of engines, see https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication/.
. (NUMBER_OF_ARGUMENTS_DOESNT_MATCH), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String&, char const* const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String&>::type, std::type_identity<char const* const&>::type>, String const&, String&, char const* const&) @ 0x000000001d0e7068
6. /build/src/Storages/MergeTree/registerStorageMergeTree.cpp:260: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000001d0e0d4b
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000193086fd
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000193086bd
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001930867d
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019308645
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
13. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
14. /build/src/Interpreters/InterpreterCreateQuery.cpp:1493: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x000000001b01ec0c
15. /build/src/Interpreters/InterpreterCreateQuery.cpp:1313: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b018b66
16. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
17. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
18. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
19. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
20. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
21. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
22. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
23. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
24. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
25. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
26. ? @ 0x00007f3c91188ac3
27. ? @ 0x00007f3c9121a850

2024.11.05 19:38:34.398597 [ 3800 ] {b8a4ee7b-c48a-436f-8493-b9e7eccef9c6} <Error> executeQuery: Code: 42. DB::Exception: With extended storage definition syntax storage ReplicatedMergeTree requires 0 to 2 parameters: 
[path in ZooKeeper],
[replica name]

Syntax for the MergeTree table engine:

CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster]
(
    name1 [type1] [DEFAULT|MATERIALIZED|ALIAS expr1] [TTL expr1],
    name2 [type2] [DEFAULT|MATERIALIZED|ALIAS expr2] [TTL expr2],
    ...
    INDEX index_name1 expr1 TYPE type1(...) [GRANULARITY value1],
    INDEX index_name2 expr2 TYPE type2(...) [GRANULARITY value2]
) ENGINE = MergeTree()
ORDER BY expr
[PARTITION BY expr]
[PRIMARY KEY expr]
[SAMPLE BY expr]
[TTL expr [DELETE|TO DISK 'xxx'|TO VOLUME 'xxx'], ...]
[SETTINGS name=value, ...]
[COMMENT 'comment']

See details in documentation: https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/. Other engines of the family support different syntax, see details in the corresponding documentation topics.

If you use the Replicated version of engines, see https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication/.
. (NUMBER_OF_ARGUMENTS_DOESNT_MATCH) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:33502) (comment: 01713_table_ttl_old_syntax_zookeeper.sql) (in query: CREATE TABLE ttl_table (`date` Date, `value` UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{database}/test_01713_table_ttl', '1', date, date, 8192) ORDER BY value SETTINGS index_granularity = 4125, min_bytes_for_wide_part = 335320873, ratio_of_defaults_for_sparse_serialization = 0., replace_long_file_name_to_hash = false, max_file_name_length = 128, min_bytes_for_full_part_storage = 536870912, compact_parts_max_bytes_to_buffer = 280561274, compact_parts_max_granules_to_buffer = 171, compact_parts_merge_max_bytes_to_prefetch_part = 6523271, merge_max_block_size = 4962, old_parts_lifetime = 337., prefer_fetch_merged_part_size_threshold = 2974024565, vertical_merge_algorithm_min_rows_to_activate = 1000000, vertical_merge_algorithm_min_columns_to_activate = 100, min_merge_bytes_to_use_direct_io = 10737418240, index_granularity_bytes = 10568046, concurrent_part_removal_threshold = 0, allow_vertical_merges_from_compact_to_wide_parts = false, cache_populated_by_fetch = false, marks_compress_block_size = 61920, primary_key_compress_block_size = 31615), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String&, char const* const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String&>::type, std::type_identity<char const* const&>::type>, String const&, String&, char const* const&) @ 0x000000001d0e7068
6. /build/src/Storages/MergeTree/registerStorageMergeTree.cpp:260: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000001d0e0d4b
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000193086fd
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000193086bd
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001930867d
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019308645
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
13. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
14. /build/src/Interpreters/InterpreterCreateQuery.cpp:1493: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x000000001b01ec0c
15. /build/src/Interpreters/InterpreterCreateQuery.cpp:1313: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b018b66
16. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
17. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
18. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
19. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
20. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
21. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
22. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
23. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
24. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
25. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
26. ? @ 0x00007f3c91188ac3
27. ? @ 0x00007f3c9121a850

2024.11.05 19:38:34.401486 [ 3800 ] {b8a4ee7b-c48a-436f-8493-b9e7eccef9c6} <Error> TCPHandler: Code: 42. DB::Exception: With extended storage definition syntax storage ReplicatedMergeTree requires 0 to 2 parameters: 
[path in ZooKeeper],
[replica name]

Syntax for the MergeTree table engine:

CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster]
(
    name1 [type1] [DEFAULT|MATERIALIZED|ALIAS expr1] [TTL expr1],
    name2 [type2] [DEFAULT|MATERIALIZED|ALIAS expr2] [TTL expr2],
    ...
    INDEX index_name1 expr1 TYPE type1(...) [GRANULARITY value1],
    INDEX index_name2 expr2 TYPE type2(...) [GRANULARITY value2]
) ENGINE = MergeTree()
ORDER BY expr
[PARTITION BY expr]
[PRIMARY KEY expr]
[SAMPLE BY expr]
[TTL expr [DELETE|TO DISK 'xxx'|TO VOLUME 'xxx'], ...]
[SETTINGS name=value, ...]
[COMMENT 'comment']

See details in documentation: https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/. Other engines of the family support different syntax, see details in the corresponding documentation topics.

If you use the Replicated version of engines, see https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication/.
. (NUMBER_OF_ARGUMENTS_DOESNT_MATCH), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String&, char const* const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String&>::type, std::type_identity<char const* const&>::type>, String const&, String&, char const* const&) @ 0x000000001d0e7068
6. /build/src/Storages/MergeTree/registerStorageMergeTree.cpp:260: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000001d0e0d4b
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000193086fd
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000193086bd
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001930867d
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019308645
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
13. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
14. /build/src/Interpreters/InterpreterCreateQuery.cpp:1493: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x000000001b01ec0c
15. /build/src/Interpreters/InterpreterCreateQuery.cpp:1313: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b018b66
16. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
17. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
18. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
19. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
20. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
21. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
22. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
23. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
24. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
25. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
26. ? @ 0x00007f3c91188ac3
27. ? @ 0x00007f3c9121a850

2024.11.05 19:38:34.413643 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:38:35.201702 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:38:37.478898 [ 3800 ] {715633e7-c378-4042-977a-5886fb76fdec} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:33506) (comment: 00159_parallel_formatting_json_and_friends_1.sh) (in query: SELECT ClientEventTime::DateTime('Asia/Dubai') as a, MobilePhoneModel as b, ClientIP6 as c FROM test.hits ORDER BY a, b, c LIMIT 3000000 Format JSONEachRow), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:38:37.497826 [ 3800 ] {715633e7-c378-4042-977a-5886fb76fdec} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:38:40.515745 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=26, jobs=0): (while reading column PageViews): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_2_2_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 0 with max_rows_to_read = 6592): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:38:41.914080 [ 2814 ] {555424bf-93ba-4578-9639-a12b5649a659} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:33534) (comment: 00159_parallel_formatting_json_and_friends_1.sh) (in query: SELECT ClientEventTime::DateTime('Asia/Dubai') as a, MobilePhoneModel as b, ClientIP6 as c FROM test.hits ORDER BY a, b, c LIMIT 3000000 Format JSONEachRow), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:38:41.945329 [ 2814 ] {555424bf-93ba-4578-9639-a12b5649a659} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:38:41.961582 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:38:45.436792 [ 3839 ] {f0ad33f2-19fd-4e68-a815-7c0b4f35ace6} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=8, jobs=8). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:34526) (comment: 00021_2_select_with_in.sql) (in query: select sum(Sign) from test.visits where CounterID in (942285, 577322);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:38:45.504488 [ 3839 ] {f0ad33f2-19fd-4e68-a815-7c0b4f35ace6} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=8, jobs=8). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:38:47.012793 [ 2814 ] {84ba677b-38f0-429a-bff5-1a2e8fd61e73} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=6). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:53914) (comment: 02028_system_data_skipping_indices_size.sql) (in query: INSERT INTO test_table VALUES ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:45: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce610d
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:487: DB::MergeTreeDataPartWriterOnDisk::fillSkipIndicesChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce4734
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:686: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3aa9
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375

2024.11.05 19:38:47.044861 [ 2814 ] {84ba677b-38f0-429a-bff5-1a2e8fd61e73} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=6). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:45: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce610d
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:487: DB::MergeTreeDataPartWriterOnDisk::fillSkipIndicesChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce4734
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:686: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3aa9
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375

2024.11.05 19:38:49.476391 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:38:55.964803 [ 3810 ] {04dca53b-99eb-4547-94da-a207b320cb88} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:53934) (comment: 00948_values_interpreter_template.sql) (in query: SELECT * FROM type_names ORDER BY n;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:38:55.978201 [ 3810 ] {04dca53b-99eb-4547-94da-a207b320cb88} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:38:59.726033 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:38:59.950915 [ 3820 ] {2de4b8e9-8571-4ac4-8dc2-a15dce9d3e85} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=7, jobs=7). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:41492) (comment: 01666_merge_tree_max_query_limit.sh) (in query: SELECT sum(read_rows) FROM system.processes WHERE query_id = 'long_running_query-test_1'), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:110: DB::CompletedPipelineExecutor::execute() @ 0x000000001d512ac8
13. /build/src/Interpreters/executeQuery.cpp:1601: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb6c20
14. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
15. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
16. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850

2024.11.05 19:39:00.034736 [ 3820 ] {2de4b8e9-8571-4ac4-8dc2-a15dce9d3e85} <Error> DynamicQueryHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=7, jobs=7). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:110: DB::CompletedPipelineExecutor::execute() @ 0x000000001d512ac8
13. /build/src/Interpreters/executeQuery.cpp:1601: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb6c20
14. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
15. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
16. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:03.935175 [ 3800 ] {e5958088-baf3-4a5b-987b-f45c51123710} <Error> executeQuery: Code: 202. DB::Exception: Too many simultaneous queries for table test_1.simple (89c899f9-9330-42af-9b72-e31183a6de3f). Maximum is: 1. (TOO_MANY_SIMULTANEOUS_QUERIES) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:50198) (comment: 01666_merge_tree_max_query_limit.sh) (in query: select * from simple), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<unsigned long&>::type>, String&&, unsigned long&) @ 0x000000000b0509ac
6. /build/src/Storages/MergeTree/MergeTreeData.cpp:8010: DB::MergeTreeData::insertQueryIdOrThrowNoLock(String const&, unsigned long) const @ 0x000000001cc3871e
7. /build/src/Storages/MergeTree/MergeTreeData.cpp:8036: DB::MergeTreeData::getQueryIdHolder(String const&, unsigned long) const @ 0x000000001cc3909d
8. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:864: DB::MergeTreeDataSelectExecutor::checkLimits(DB::MergeTreeData const&, DB::ReadFromMergeTree::AnalysisResult const&, std::shared_ptr<DB::Context const> const&) @ 0x000000001cd13406
9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:2005: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd63c46
10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
13. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
14. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
15. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
16. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
17. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
18. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
19. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
20. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
21. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
22. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
23. ? @ 0x00007f3c91188ac3
24. ? @ 0x00007f3c9121a850

2024.11.05 19:39:03.977349 [ 3800 ] {e5958088-baf3-4a5b-987b-f45c51123710} <Error> TCPHandler: Code: 202. DB::Exception: Too many simultaneous queries for table test_1.simple (89c899f9-9330-42af-9b72-e31183a6de3f). Maximum is: 1. (TOO_MANY_SIMULTANEOUS_QUERIES), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<unsigned long&>::type>, String&&, unsigned long&) @ 0x000000000b0509ac
6. /build/src/Storages/MergeTree/MergeTreeData.cpp:8010: DB::MergeTreeData::insertQueryIdOrThrowNoLock(String const&, unsigned long) const @ 0x000000001cc3871e
7. /build/src/Storages/MergeTree/MergeTreeData.cpp:8036: DB::MergeTreeData::getQueryIdHolder(String const&, unsigned long) const @ 0x000000001cc3909d
8. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:864: DB::MergeTreeDataSelectExecutor::checkLimits(DB::MergeTreeData const&, DB::ReadFromMergeTree::AnalysisResult const&, std::shared_ptr<DB::Context const> const&) @ 0x000000001cd13406
9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:2005: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd63c46
10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
13. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
14. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
15. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
16. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
17. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
18. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
19. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
20. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
21. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
22. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
23. ? @ 0x00007f3c91188ac3
24. ? @ 0x00007f3c9121a850

2024.11.05 19:39:05.110599 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {c04f5a1b-d832-4de4-96b0-dec4e591ea83::202411_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:274: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3abc4
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:06.125761 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:10.189188 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:18.520755 [ 3839 ] {e6815a86-93a9-407f-9249-3657c9b8cada} <Error> executeQuery: Code: 202. DB::Exception: Too many simultaneous queries for table test_1.simple (89c899f9-9330-42af-9b72-e31183a6de3f). Maximum is: 1. (TOO_MANY_SIMULTANEOUS_QUERIES) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52262) (comment: 01666_merge_tree_max_query_limit.sh) (in query: select * from simple where i = 0), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<unsigned long&>::type>, String&&, unsigned long&) @ 0x000000000b0509ac
6. /build/src/Storages/MergeTree/MergeTreeData.cpp:8010: DB::MergeTreeData::insertQueryIdOrThrowNoLock(String const&, unsigned long) const @ 0x000000001cc3871e
7. /build/src/Storages/MergeTree/MergeTreeData.cpp:8036: DB::MergeTreeData::getQueryIdHolder(String const&, unsigned long) const @ 0x000000001cc3909d
8. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:864: DB::MergeTreeDataSelectExecutor::checkLimits(DB::MergeTreeData const&, DB::ReadFromMergeTree::AnalysisResult const&, std::shared_ptr<DB::Context const> const&) @ 0x000000001cd13406
9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:2005: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd63c46
10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
13. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
14. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
15. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
16. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
17. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
18. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
19. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
20. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
21. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
22. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
23. ? @ 0x00007f3c91188ac3
24. ? @ 0x00007f3c9121a850

2024.11.05 19:39:18.579051 [ 3839 ] {e6815a86-93a9-407f-9249-3657c9b8cada} <Error> TCPHandler: Code: 202. DB::Exception: Too many simultaneous queries for table test_1.simple (89c899f9-9330-42af-9b72-e31183a6de3f). Maximum is: 1. (TOO_MANY_SIMULTANEOUS_QUERIES), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<unsigned long&>::type>, String&&, unsigned long&) @ 0x000000000b0509ac
6. /build/src/Storages/MergeTree/MergeTreeData.cpp:8010: DB::MergeTreeData::insertQueryIdOrThrowNoLock(String const&, unsigned long) const @ 0x000000001cc3871e
7. /build/src/Storages/MergeTree/MergeTreeData.cpp:8036: DB::MergeTreeData::getQueryIdHolder(String const&, unsigned long) const @ 0x000000001cc3909d
8. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:864: DB::MergeTreeDataSelectExecutor::checkLimits(DB::MergeTreeData const&, DB::ReadFromMergeTree::AnalysisResult const&, std::shared_ptr<DB::Context const> const&) @ 0x000000001cd13406
9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:2005: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd63c46
10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
13. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
14. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
15. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
16. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
17. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
18. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
19. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
20. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
21. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
22. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
23. ? @ 0x00007f3c91188ac3
24. ? @ 0x00007f3c9121a850

2024.11.05 19:39:18.662448 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:26.761292 [ 3820 ] {895e1037-7b39-45e4-bef2-13017c6794c1} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=81): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:50216) (comment: 00164_quantileBfloat16.sql) (in query: SELECT CounterID AS k, quantileBFloat16(0.5)(ResolutionWidth) FROM test.hits GROUP BY k ORDER BY count() DESC, CounterID LIMIT 10;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:39:26.766075 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:27.187201 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:27.930523 [ 3641 ] {} <Error> void DB::SystemLog<DB::PartLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::PartLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:28.780266 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:29.337633 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:31.324002 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 1 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:32.161202 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:33.040000 [ 3820 ] {895e1037-7b39-45e4-bef2-13017c6794c1} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=81): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:39:33.226159 [ 3643 ] {} <Error> void DB::SystemLog<DB::SessionLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::SessionLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:34.748369 [ 4209 ] {} <Error> void DB::AsyncLoader::worker(Pool &): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=908, jobs=824). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:34.864034 [ 4592 ] {} <Error> DDLWorker(test_i58sjq98): Error on initialization of test_i58sjq98: Code: 695. DB::Exception: Load job 'startup Replicated database test_i58sjq98' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=908, jobs=824). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:35.417235 [ 2816 ] {c63b7283-64e9-4402-a049-fe66efce918f} <Error> executeQuery: Code: 202. DB::Exception: Too many simultaneous queries for table test_1.simple (89c899f9-9330-42af-9b72-e31183a6de3f). Maximum is: 1. (TOO_MANY_SIMULTANEOUS_QUERIES) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:39958) (comment: 01666_merge_tree_max_query_limit.sh) (in query: select * from simple where i = 0), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<unsigned long&>::type>, String&&, unsigned long&) @ 0x000000000b0509ac
6. /build/src/Storages/MergeTree/MergeTreeData.cpp:8010: DB::MergeTreeData::insertQueryIdOrThrowNoLock(String const&, unsigned long) const @ 0x000000001cc3871e
7. /build/src/Storages/MergeTree/MergeTreeData.cpp:8036: DB::MergeTreeData::getQueryIdHolder(String const&, unsigned long) const @ 0x000000001cc3909d
8. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:864: DB::MergeTreeDataSelectExecutor::checkLimits(DB::MergeTreeData const&, DB::ReadFromMergeTree::AnalysisResult const&, std::shared_ptr<DB::Context const> const&) @ 0x000000001cd13406
9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:2005: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd63c46
10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
13. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
14. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
15. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
16. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
17. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
18. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
19. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
20. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
21. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
22. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
23. ? @ 0x00007f3c91188ac3
24. ? @ 0x00007f3c9121a850

2024.11.05 19:39:35.450578 [ 2816 ] {c63b7283-64e9-4402-a049-fe66efce918f} <Error> TCPHandler: Code: 202. DB::Exception: Too many simultaneous queries for table test_1.simple (89c899f9-9330-42af-9b72-e31183a6de3f). Maximum is: 1. (TOO_MANY_SIMULTANEOUS_QUERIES), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<unsigned long&>::type>, String&&, unsigned long&) @ 0x000000000b0509ac
6. /build/src/Storages/MergeTree/MergeTreeData.cpp:8010: DB::MergeTreeData::insertQueryIdOrThrowNoLock(String const&, unsigned long) const @ 0x000000001cc3871e
7. /build/src/Storages/MergeTree/MergeTreeData.cpp:8036: DB::MergeTreeData::getQueryIdHolder(String const&, unsigned long) const @ 0x000000001cc3909d
8. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:864: DB::MergeTreeDataSelectExecutor::checkLimits(DB::MergeTreeData const&, DB::ReadFromMergeTree::AnalysisResult const&, std::shared_ptr<DB::Context const> const&) @ 0x000000001cd13406
9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:2005: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd63c46
10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
13. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
14. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
15. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
16. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
17. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
18. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
19. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
20. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
21. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
22. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
23. ? @ 0x00007f3c91188ac3
24. ? @ 0x00007f3c9121a850

2024.11.05 19:39:38.490339 [ 3810 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/eko/xvngvgcxobzsstdjctxshpnrnijgp, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b28a4648-d857-414a-862a-482b128e9ec8, upload has finished false.
2024.11.05 19:39:38.492297 [ 3810 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/eko/xvngvgcxobzsstdjctxshpnrnijgp, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b28a4648-d857-414a-862a-482b128e9ec8, upload has finished false
2024.11.05 19:39:38.601822 [ 3810 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/eko/xvngvgcxobzsstdjctxshpnrnijgp, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b28a4648-d857-414a-862a-482b128e9ec8, upload has finished false
2024.11.05 19:39:38.620851 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:38.742139 [ 3810 ] {e6d2c5eb-816d-4af3-9a21-922ea4cc691b} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 15 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:53026) (comment: 02725_memory-for-merges.sql) (in query: OPTIMIZE TABLE 02725_memory_for_merges FINAL;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110

2024.11.05 19:39:38.772794 [ 3810 ] {e6d2c5eb-816d-4af3-9a21-922ea4cc691b} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 15 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110

2024.11.05 19:39:40.411664 [ 4592 ] {} <Error> DDLWorker(test_i58sjq98): Error on initialization of test_i58sjq98: Code: 695. DB::Exception: Load job 'startup Replicated database test_i58sjq98' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=908, jobs=824). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:40.412066 [ 2814 ] {long_running_query-test_1} <Error> executeQuery: Code: 394. DB::Exception: Query was cancelled. (QUERY_WAS_CANCELLED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:53046) (comment: 01666_merge_tree_max_query_limit.sh) (in query: select sleepEachRow(0.1) from simple settings max_block_size = 1 format Null), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Interpreters/ProcessList.cpp:480: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c05
7. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
8. /build/src/Processors/Executors/PipelineExecutor.cpp:196: DB::PipelineExecutor::finalizeExecution() @ 0x000000001d515c7f
9. /build/src/Processors/Executors/PipelineExecutor.cpp:131: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d51553e
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
12. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
13. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
14. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
15. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
22. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
23. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
25. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
26. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
27. ? @ 0x00007f3c91188ac3
28. ? @ 0x00007f3c9121a850

2024.11.05 19:39:40.508409 [ 2814 ] {long_running_query-test_1} <Error> TCPHandler: Code: 394. DB::Exception: Query was cancelled. (QUERY_WAS_CANCELLED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Interpreters/ProcessList.cpp:480: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c05
7. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
8. /build/src/Processors/Executors/PipelineExecutor.cpp:196: DB::PipelineExecutor::finalizeExecution() @ 0x000000001d515c7f
9. /build/src/Processors/Executors/PipelineExecutor.cpp:131: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d51553e
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
12. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
13. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
14. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
15. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
22. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
23. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
25. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
26. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
27. ? @ 0x00007f3c91188ac3
28. ? @ 0x00007f3c9121a850

2024.11.05 19:39:45.406811 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/vot/acgmofguisaplihfjapepljlwqoba, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 340209ed-118c-493b-ba46-7d85bbfa7a26, upload has finished false.
2024.11.05 19:39:45.407351 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/vot/acgmofguisaplihfjapepljlwqoba, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 340209ed-118c-493b-ba46-7d85bbfa7a26, upload has finished false
2024.11.05 19:39:45.418613 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/vot/acgmofguisaplihfjapepljlwqoba, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 340209ed-118c-493b-ba46-7d85bbfa7a26, upload has finished false
2024.11.05 19:39:45.471627 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 12 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:45.488269 [ 4592 ] {} <Error> DDLWorker(test_i58sjq98): Error on initialization of test_i58sjq98: Code: 695. DB::Exception: Load job 'startup Replicated database test_i58sjq98' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=908, jobs=824). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:46.936807 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:50.512518 [ 4592 ] {} <Error> DDLWorker(test_i58sjq98): Error on initialization of test_i58sjq98: Code: 695. DB::Exception: Load job 'startup Replicated database test_i58sjq98' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=908, jobs=824). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:51.146189 [ 2814 ] {f3cee040-e19b-4fc6-a465-3d78806f9662} <Error> executeQuery: Code: 248. DB::Exception: Invalid partition format: NO_SUCH_PART. (INVALID_PARTITION_VALUE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:59386) (comment: 02004_invalid_partition_mutation_stuck.sql) (in query: ALTER TABLE rep_data MATERIALIZE INDEX idx IN PARTITION ID 'NO_SUCH_PART';), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Storages/MergeTree/MergeTreePartInfo.cpp:50: DB::MergeTreePartInfo::validatePartitionID(std::shared_ptr<DB::IAST> const&, StrongTypedef<unsigned int, DB::MergeTreeDataFormatVersionTag>) @ 0x000000001ce293d1
7. /build/src/Storages/MergeTree/MergeTreeData.cpp:5677: DB::MergeTreeData::getPartitionIDFromQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, DB::DataPartsLock*) const @ 0x000000001cbedb30
8. /build/src/Storages/MergeTree/MergeTreeData.cpp:5924: DB::MergeTreeData::getPartitionIdsAffectedByCommands(DB::MutationCommands const&, std::shared_ptr<DB::Context const>) const @ 0x000000001cc15ddd
9. /build/src/Storages/StorageReplicatedMergeTree.cpp:5969: DB::StorageReplicatedMergeTree::allocateBlockNumbersInAffectedPartitions(DB::MutationCommands const&, std::shared_ptr<DB::Context const>, std::shared_ptr<zkutil::ZooKeeper> const&) const @ 0x000000001c54ef16
10. /build/src/Storages/StorageReplicatedMergeTree.cpp:7456: DB::StorageReplicatedMergeTree::mutate(DB::MutationCommands const&, std::shared_ptr<DB::Context const>) @ 0x000000001c5743f7
11. /build/src/Interpreters/InterpreterAlterQuery.cpp:221: DB::InterpreterAlterQuery::executeToTable(DB::ASTAlterQuery const&) @ 0x000000001affb000
12. /build/src/Interpreters/InterpreterAlterQuery.cpp:67: DB::InterpreterAlterQuery::execute() @ 0x000000001aff84de
13. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
14. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
15. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
16. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850

2024.11.05 19:39:51.153039 [ 2814 ] {f3cee040-e19b-4fc6-a465-3d78806f9662} <Error> TCPHandler: Code: 248. DB::Exception: Invalid partition format: NO_SUCH_PART. (INVALID_PARTITION_VALUE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Storages/MergeTree/MergeTreePartInfo.cpp:50: DB::MergeTreePartInfo::validatePartitionID(std::shared_ptr<DB::IAST> const&, StrongTypedef<unsigned int, DB::MergeTreeDataFormatVersionTag>) @ 0x000000001ce293d1
7. /build/src/Storages/MergeTree/MergeTreeData.cpp:5677: DB::MergeTreeData::getPartitionIDFromQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, DB::DataPartsLock*) const @ 0x000000001cbedb30
8. /build/src/Storages/MergeTree/MergeTreeData.cpp:5924: DB::MergeTreeData::getPartitionIdsAffectedByCommands(DB::MutationCommands const&, std::shared_ptr<DB::Context const>) const @ 0x000000001cc15ddd
9. /build/src/Storages/StorageReplicatedMergeTree.cpp:5969: DB::StorageReplicatedMergeTree::allocateBlockNumbersInAffectedPartitions(DB::MutationCommands const&, std::shared_ptr<DB::Context const>, std::shared_ptr<zkutil::ZooKeeper> const&) const @ 0x000000001c54ef16
10. /build/src/Storages/StorageReplicatedMergeTree.cpp:7456: DB::StorageReplicatedMergeTree::mutate(DB::MutationCommands const&, std::shared_ptr<DB::Context const>) @ 0x000000001c5743f7
11. /build/src/Interpreters/InterpreterAlterQuery.cpp:221: DB::InterpreterAlterQuery::executeToTable(DB::ASTAlterQuery const&) @ 0x000000001affb000
12. /build/src/Interpreters/InterpreterAlterQuery.cpp:67: DB::InterpreterAlterQuery::execute() @ 0x000000001aff84de
13. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
14. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
15. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
16. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850

2024.11.05 19:39:52.162737 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:52.636968 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/ckp/ulcaswsadtfppdpwxdcgrvaldbnsz, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ffb0df9e-a768-467e-8eae-845d3a6ed984, upload has finished false.
2024.11.05 19:39:52.639626 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/ckp/ulcaswsadtfppdpwxdcgrvaldbnsz, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ffb0df9e-a768-467e-8eae-845d3a6ed984, upload has finished false
2024.11.05 19:39:52.683454 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/ckp/ulcaswsadtfppdpwxdcgrvaldbnsz, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ffb0df9e-a768-467e-8eae-845d3a6ed984, upload has finished false
2024.11.05 19:39:52.687707 [ 3610 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Removing temporary directory /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:39:52.689218 [ 3610 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:39:52.746071 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 15 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:54.503781 [ 2814 ] {44e931a4-677a-4309-9f82-7a06cf959a55} <Error> executeQuery: Code: 248. DB::Exception: Invalid partition format: NO_SUCH_PART. (INVALID_PARTITION_VALUE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:59386) (comment: 02004_invalid_partition_mutation_stuck.sql) (in query: ALTER TABLE data MATERIALIZE INDEX idx IN PARTITION ID 'NO_SUCH_PART';), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Storages/MergeTree/MergeTreePartInfo.cpp:50: DB::MergeTreePartInfo::validatePartitionID(std::shared_ptr<DB::IAST> const&, StrongTypedef<unsigned int, DB::MergeTreeDataFormatVersionTag>) @ 0x000000001ce293d1
7. /build/src/Storages/MergeTree/MergeTreeData.cpp:5677: DB::MergeTreeData::getPartitionIDFromQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, DB::DataPartsLock*) const @ 0x000000001cbedb30
8. /build/src/Storages/MergeTree/MergeTreeData.cpp:5924: DB::MergeTreeData::getPartitionIdsAffectedByCommands(DB::MutationCommands const&, std::shared_ptr<DB::Context const>) const @ 0x000000001cc15ddd
9. /build/src/Storages/StorageMergeTree.cpp:644: DB::StorageMergeTree::mutate(DB::MutationCommands const&, std::shared_ptr<DB::Context const>) @ 0x000000001d0f4f00
10. /build/src/Interpreters/InterpreterAlterQuery.cpp:221: DB::InterpreterAlterQuery::executeToTable(DB::ASTAlterQuery const&) @ 0x000000001affb000
11. /build/src/Interpreters/InterpreterAlterQuery.cpp:67: DB::InterpreterAlterQuery::execute() @ 0x000000001aff84de
12. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
13. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
14. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
15. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:39:54.509787 [ 2814 ] {44e931a4-677a-4309-9f82-7a06cf959a55} <Error> TCPHandler: Code: 248. DB::Exception: Invalid partition format: NO_SUCH_PART. (INVALID_PARTITION_VALUE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Storages/MergeTree/MergeTreePartInfo.cpp:50: DB::MergeTreePartInfo::validatePartitionID(std::shared_ptr<DB::IAST> const&, StrongTypedef<unsigned int, DB::MergeTreeDataFormatVersionTag>) @ 0x000000001ce293d1
7. /build/src/Storages/MergeTree/MergeTreeData.cpp:5677: DB::MergeTreeData::getPartitionIDFromQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, DB::DataPartsLock*) const @ 0x000000001cbedb30
8. /build/src/Storages/MergeTree/MergeTreeData.cpp:5924: DB::MergeTreeData::getPartitionIdsAffectedByCommands(DB::MutationCommands const&, std::shared_ptr<DB::Context const>) const @ 0x000000001cc15ddd
9. /build/src/Storages/StorageMergeTree.cpp:644: DB::StorageMergeTree::mutate(DB::MutationCommands const&, std::shared_ptr<DB::Context const>) @ 0x000000001d0f4f00
10. /build/src/Interpreters/InterpreterAlterQuery.cpp:221: DB::InterpreterAlterQuery::executeToTable(DB::ASTAlterQuery const&) @ 0x000000001affb000
11. /build/src/Interpreters/InterpreterAlterQuery.cpp:67: DB::InterpreterAlterQuery::execute() @ 0x000000001aff84de
12. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
13. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
14. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
15. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:39:54.601361 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:55.543333 [ 4592 ] {} <Error> DDLWorker(test_i58sjq98): Error on initialization of test_i58sjq98: Code: 695. DB::Exception: Load job 'startup Replicated database test_i58sjq98' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=908, jobs=824). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:39:57.234290 [ 2816 ] {5c3128f6-7c9e-4b67-a395-51b23b1250ce} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:39968) (comment: 00153_aggregate_arena_race.sql) (in query: insert into dest00153 select groupUniqArrayState(RefererDomain) from test.hits group by URLDomain;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:40:00.493599 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/afk/cmvvfjjpykqphutvkhxmobvxtltdn, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6cbac274-a756-4647-a5f3-959e29ec02cb, upload has finished false.
2024.11.05 19:40:00.495070 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/afk/cmvvfjjpykqphutvkhxmobvxtltdn, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6cbac274-a756-4647-a5f3-959e29ec02cb, upload has finished false
2024.11.05 19:40:00.572910 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/afk/cmvvfjjpykqphutvkhxmobvxtltdn, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6cbac274-a756-4647-a5f3-959e29ec02cb, upload has finished false
2024.11.05 19:40:00.615344 [ 4592 ] {} <Error> DDLWorker(test_i58sjq98): Error on initialization of test_i58sjq98: Code: 695. DB::Exception: Load job 'startup Replicated database test_i58sjq98' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=908, jobs=824). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:00.673272 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 15 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:01.834483 [ 2816 ] {5c3128f6-7c9e-4b67-a395-51b23b1250ce} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:40:04.589970 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:06.884649 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/amj/cqgnljqsyxbvypwztcftpfazakhgp, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 81ff608f-4e4c-4552-b81b-002d4c3ca8d5, upload has finished false.
2024.11.05 19:40:06.885637 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/amj/cqgnljqsyxbvypwztcftpfazakhgp, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 81ff608f-4e4c-4552-b81b-002d4c3ca8d5, upload has finished false
2024.11.05 19:40:06.920077 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/amj/cqgnljqsyxbvypwztcftpfazakhgp, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 81ff608f-4e4c-4552-b81b-002d4c3ca8d5, upload has finished false
2024.11.05 19:40:07.027678 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 13 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:09.061365 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 2 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:09.204287 [ 3641 ] {} <Error> void DB::SystemLog<DB::PartLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::PartLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:09.782260 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:09.795925 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column StartDate): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_5_5_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 0 with max_rows_to_read = 6739): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:11.397538 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 2 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:12.597394 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:12.948479 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 1 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:12.965327 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:14.652198 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=8). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:17.059450 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:17.362469 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column VisitID): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_3_3_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 1 with max_rows_to_read = 6608): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:18.013824 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:19.244875 [ 3642 ] {} <Error> void DB::SystemLog<DB::MetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::MetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:22.508271 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/yns/mehzgaeaxgphcpyboeicdbqnojlzy, total size 100663296, count 100663296, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id e7810f51-1506-4ddb-be6f-4530ee0c77e2, upload has finished false.
2024.11.05 19:40:22.512598 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/yns/mehzgaeaxgphcpyboeicdbqnojlzy, total size 100663296, count 100663296, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id e7810f51-1506-4ddb-be6f-4530ee0c77e2, upload has finished false
2024.11.05 19:40:22.579467 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/yns/mehzgaeaxgphcpyboeicdbqnojlzy, total size 100663296, count 100663296, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id e7810f51-1506-4ddb-be6f-4530ee0c77e2, upload has finished false
2024.11.05 19:40:22.812186 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 23 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:23.575276 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_16_3}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:24.277837 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:26.760469 [ 2814 ] {0b79330b-c99a-4909-94ce-094d60567a00} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:36646) (comment: 00050_min_max.sql) (in query: SELECT CounterID, min(WatchID), max(WatchID) FROM test.hits GROUP BY CounterID ORDER BY count() DESC LIMIT 20;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:40:27.103205 [ 2814 ] {0b79330b-c99a-4909-94ce-094d60567a00} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:40:28.488416 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/bhy/mlbiplbcpwkwhtroentomzlwgpivd, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c0783f1e-2932-486f-910b-5a085d6a9dc1, upload has finished false.
2024.11.05 19:40:28.492907 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/bhy/mlbiplbcpwkwhtroentomzlwgpivd, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c0783f1e-2932-486f-910b-5a085d6a9dc1, upload has finished false
2024.11.05 19:40:28.533866 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/bhy/mlbiplbcpwkwhtroentomzlwgpivd, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c0783f1e-2932-486f-910b-5a085d6a9dc1, upload has finished false
2024.11.05 19:40:28.571488 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:567: DB::WriteBufferFromS3::writePart(DB::WriteBufferFromS3::PartData&&) @ 0x0000000019914d6e
18. /build/src/IO/WriteBufferFromS3.cpp:370: DB::WriteBufferFromS3::writeMultipartUpload() @ 0x0000000019908eb6
19. /build/src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::nextImpl() @ 0x000000001990857e
20. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
21. /build/src/IO/WriteBufferFromFileDecorator.cpp:71: DB::WriteBufferFromFileDecorator::nextImpl() @ 0x000000001a3593ad
22. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
23. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
24. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
25. /build/src/IO/WriteBuffer.h:76: DB::WriteBuffer::nextIfAtEnd() @ 0x000000000a931978
26. /build/src/IO/WriteBuffer.h:91: DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000a9317e0
27. /build/src/Compression/CompressedWriteBuffer.cpp:58: DB::CompressedWriteBuffer::nextImpl() @ 0x0000000019ddd7ed
28. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
29. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
30. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
31. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:350: DB::MergeTreeDataPartWriterWide::getCurrentMarksForColumn(DB::NameAndTypePair const&, std::set<String, std::less<String>, std::allocator<String>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ccf5839
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:33.213802 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:35.329589 [ 3583 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/djm/ibilfovrxcgnvntlohcxiiseixhzs, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 1d0439a5-7d02-421e-88c1-a49d3c4787d4, upload has finished false.
2024.11.05 19:40:35.330271 [ 3583 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/djm/ibilfovrxcgnvntlohcxiiseixhzs, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 1d0439a5-7d02-421e-88c1-a49d3c4787d4, upload has finished false
2024.11.05 19:40:35.353841 [ 3583 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/djm/ibilfovrxcgnvntlohcxiiseixhzs, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 1d0439a5-7d02-421e-88c1-a49d3c4787d4, upload has finished false
2024.11.05 19:40:35.409190 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 16 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:36.514281 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:40.044914 [ 2814 ] {b94135aa-f6f1-496c-85cf-3c0726a30ec3} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:49502) (comment: 00059_merge_sorting_empty_array_joined.sql) (in query: SELECT CounterID FROM test.visits ARRAY JOIN Goals.ID WHERE CounterID = 942285 ORDER BY CounterID ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:40:40.071144 [ 2814 ] {b94135aa-f6f1-496c-85cf-3c0726a30ec3} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:40:40.204456 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 7 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:46.799684 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {dbd666d3-2d23-40c6-8136-6ca2be336b4d::202411_1_16_3}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
24. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
25. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
26. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
27. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:51.243195 [ 2814 ] {52fa70ee-2272-4142-8fcb-91cfdb10c7f5} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:42016) (comment: 00653_verification_monotonic_data_load.sh) (in query: CREATE TABLE unsigned_integer_test_table (val UInt32) ENGINE = MergeTree ORDER BY val SETTINGS index_granularity = 1, index_granularity_bytes = 0, min_bytes_for_wide_part = 0;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/IO/WriteBufferFromFileDecorator.cpp:0: DB::WriteBufferFromFileDecorator::finalizeImpl() @ 0x000000001a359047
21. /build/src/Disks/IO/WriteBufferWithFinalizeCallback.cpp:18: DB::WriteBufferWithFinalizeCallback::finalizeImpl() @ 0x000000001a358db9
22. /build/src/IO/WriteBuffer.h:127: DB::WriteBuffer::finalize() @ 0x000000000a929062
23. /build/src/Storages/MergeTree/MergeTreeData.cpp:306: DB::MergeTreeData::initializeDirectoriesAndFormatVersion(String const&, bool, String const&, bool) @ 0x000000001cbae542
24. /build/src/Storages/StorageMergeTree.cpp:117: DB::StorageMergeTree::StorageMergeTree(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata const&, DB::LoadingStrictnessLevel, std::shared_ptr<DB::Context>, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>) @ 0x000000001d0eb9d1
25. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageMergeTree* std::construct_at[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::StorageMergeTree*>(DB::StorageMergeTree*, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eb3b7
26. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageMergeTree>>::construct[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void, void>(std::allocator<DB::StorageMergeTree>&, DB::StorageMergeTree*, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eb0c7
27. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageMergeTree, std::allocator<DB::StorageMergeTree>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>>(std::allocator<DB::StorageMergeTree>, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0ead75
28. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageMergeTree> std::allocate_shared[abi:v15000]<DB::StorageMergeTree, std::allocator<DB::StorageMergeTree>, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void>(std::allocator<DB::StorageMergeTree> const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eaba7
29. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageMergeTree> std::make_shared[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void>(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0e7d6d
30. /build/src/Storages/MergeTree/registerStorageMergeTree.cpp:741: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000001d0e58aa
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000193086fd

2024.11.05 19:40:51.546992 [ 2814 ] {52fa70ee-2272-4142-8fcb-91cfdb10c7f5} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/IO/WriteBufferFromFileDecorator.cpp:0: DB::WriteBufferFromFileDecorator::finalizeImpl() @ 0x000000001a359047
21. /build/src/Disks/IO/WriteBufferWithFinalizeCallback.cpp:18: DB::WriteBufferWithFinalizeCallback::finalizeImpl() @ 0x000000001a358db9
22. /build/src/IO/WriteBuffer.h:127: DB::WriteBuffer::finalize() @ 0x000000000a929062
23. /build/src/Storages/MergeTree/MergeTreeData.cpp:306: DB::MergeTreeData::initializeDirectoriesAndFormatVersion(String const&, bool, String const&, bool) @ 0x000000001cbae542
24. /build/src/Storages/StorageMergeTree.cpp:117: DB::StorageMergeTree::StorageMergeTree(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata const&, DB::LoadingStrictnessLevel, std::shared_ptr<DB::Context>, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>) @ 0x000000001d0eb9d1
25. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageMergeTree* std::construct_at[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::StorageMergeTree*>(DB::StorageMergeTree*, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eb3b7
26. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageMergeTree>>::construct[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void, void>(std::allocator<DB::StorageMergeTree>&, DB::StorageMergeTree*, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eb0c7
27. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageMergeTree, std::allocator<DB::StorageMergeTree>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>>(std::allocator<DB::StorageMergeTree>, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0ead75
28. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageMergeTree> std::allocate_shared[abi:v15000]<DB::StorageMergeTree, std::allocator<DB::StorageMergeTree>, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void>(std::allocator<DB::StorageMergeTree> const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eaba7
29. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageMergeTree> std::make_shared[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void>(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0e7d6d
30. /build/src/Storages/MergeTree/registerStorageMergeTree.cpp:741: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000001d0e58aa
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000193086fd

2024.11.05 19:40:51.850387 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:45: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce610d
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
23. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
24. /build/src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp:69: DB::MergedColumnOnlyOutputStream::fillChecksums(std::shared_ptr<DB::IMergeTreeDataPart>&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3fe1a
25. /build/src/Storages/MergeTree/MergeTask.cpp:657: DB::MergeTask::VerticalMergeStage::finalizeVerticalMergeForOneColumn() const @ 0x000000001cb6fabd
26. /build/src/Storages/MergeTree/MergeTask.cpp:877: DB::MergeTask::VerticalMergeStage::executeVerticalMergeForAllColumns() const @ 0x000000001cb72f6e
27. /build/src/Storages/MergeTree/MergeTask.cpp:829: DB::MergeTask::VerticalMergeStage::execute() @ 0x000000001cb72cb8
28. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
29. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:51.953363 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:52.553940 [ 3583 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/jrw/abfycqvkeamopkdenfnpizvbmtbwg, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8d37d6f6-4dc9-434d-86ee-2d5b4ffa0fc8, upload has finished false.
2024.11.05 19:40:52.561043 [ 3583 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/jrw/abfycqvkeamopkdenfnpizvbmtbwg, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8d37d6f6-4dc9-434d-86ee-2d5b4ffa0fc8, upload has finished false
2024.11.05 19:40:52.635529 [ 3583 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/jrw/abfycqvkeamopkdenfnpizvbmtbwg, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8d37d6f6-4dc9-434d-86ee-2d5b4ffa0fc8, upload has finished false
2024.11.05 19:40:52.717444 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 35 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:53.837028 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:54.584352 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:55.112114 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {dbd666d3-2d23-40c6-8136-6ca2be336b4d::202411_1_16_3}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:56.754481 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=1): (while reading column UserID): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_1_1_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 1 with max_rows_to_read = 6658): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:57.331937 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 6 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:57.965524 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=9). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:40:58.543874 [ 3800 ] {20dc56ed-9f2d-4567-a090-0fdf19c0a8bc} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=22): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:49458) (comment: 00159_parallel_formatting_json_and_friends_1.sh) (in query: SELECT ClientEventTime::DateTime('Asia/Dubai') as a, MobilePhoneModel as b, ClientIP6 as c FROM test.hits ORDER BY a, b, c LIMIT 3000000 Format JSONCompactStringsEachRow), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:40:59.305385 [ 3800 ] {20dc56ed-9f2d-4567-a090-0fdf19c0a8bc} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=22): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:41:00.688593 [ 2816 ] {3ab6b072-ea9c-4bf2-9b8d-f34669192ba2} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=12, jobs=12). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:41990) (comment: 00170_s3_cache.sql) (in query: SELECT count() FROM test.hits_s3;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:41:00.702009 [ 2816 ] {3ab6b072-ea9c-4bf2-9b8d-f34669192ba2} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=12, jobs=12). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:41:04.201985 [ 3612 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Removing temporary directory /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:41:04.202588 [ 3612 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:41:04.257643 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/mpr/essfdrheldzddztrhiltyorvxuxho, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 12e0967b-8b04-4c29-bf7c-a1078ad9ef8a, upload has finished false.
2024.11.05 19:41:04.264501 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/mpr/essfdrheldzddztrhiltyorvxuxho, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 12e0967b-8b04-4c29-bf7c-a1078ad9ef8a, upload has finished false
2024.11.05 19:41:04.320851 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/mpr/essfdrheldzddztrhiltyorvxuxho, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 12e0967b-8b04-4c29-bf7c-a1078ad9ef8a, upload has finished false
2024.11.05 19:41:04.465318 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 16 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:05.733064 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:06.107015 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:08.744742 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:09.383632 [ 2816 ] {5a33c876-86ab-42b1-98bc-7dafe013df43} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:40528) (comment: 00012_sorting_distributed.sql) (in query: SELECT CAST(`__table1`.`EventTime`, 'DateTime(\'Asia/Dubai\')') AS `CAST(EventTime, 'DateTime(\\'Asia/Dubai\\')')` FROM `test`.`hits` AS `__table1` ORDER BY `__table1`.`EventTime` DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:41:09.871506 [ 2816 ] {5a33c876-86ab-42b1-98bc-7dafe013df43} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:41:10.152367 [ 2814 ] {be93a83a-9a3b-4653-be0b-d047544f0b33} <Error> executeQuery: Code: 439. DB::Exception: Received from 127.0.0.1:9000. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). Stack trace:

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
: While executing Remote. (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:45104) (comment: 00012_sorting_distributed.sql) (in query: SELECT EventTime::DateTime('Asia/Dubai') FROM remote('127.0.0.{1,2}', test, hits) ORDER BY EventTime DESC LIMIT 10 ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:89: DB::Exception::Exception(String const&, int, bool) @ 0x000000000a9434aa
5. /build/src/Common/Exception.h:97: DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000a943455
6. /build/src/IO/ReadHelpers.cpp:1560: DB::readException(DB::ReadBuffer&, String const&, bool) @ 0x000000001336a782
7. /build/src/Client/Connection.cpp:1233: DB::Connection::receiveException() const @ 0x000000001d2be459
8. /build/src/Client/Connection.cpp:1070: DB::Connection::receivePacket() @ 0x000000001d2c59b1
9. /build/src/Client/PacketReceiver.cpp:66: DB::PacketReceiver::Task::run(std::function<void (int, Poco::Timespan, DB::AsyncEventTimeoutType, String const&, unsigned int)>, std::function<void ()>) @ 0x000000001d3328a4
10. /build/src/Common/AsyncTaskExecutor.cpp:88: DB::AsyncTaskExecutor::Routine::operator()(std::function<void ()>) @ 0x0000000019ebe669
11. /build/src/Common/Fiber.h:76: Fiber::RoutineImpl<DB::AsyncTaskExecutor::Routine>::operator()(boost::context::fiber&&) @ 0x0000000019ebe578
12. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x0000000019ebe508
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:531: ? @ 0x0000000019ebe4c8
14. /build/contrib/boost/boost/context/fiber_fcontext.hpp:157: boost::context::detail::fiber_record<boost::context::fiber, FiberStack&, Fiber::RoutineImpl<DB::AsyncTaskExecutor::Routine>>::run(void*) @ 0x0000000019ebe3fe
15. /build/contrib/boost/boost/context/fiber_fcontext.hpp:97: void boost::context::detail::fiber_entry<boost::context::detail::fiber_record<boost::context::fiber, FiberStack&, Fiber::RoutineImpl<DB::AsyncTaskExecutor::Routine>>>(boost::context::detail::transfer_t) @ 0x0000000019ebe11d

2024.11.05 19:41:10.255418 [ 2814 ] {be93a83a-9a3b-4653-be0b-d047544f0b33} <Error> TCPHandler: Code: 439. DB::Exception: Received from 127.0.0.1:9000. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). Stack trace:

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
: While executing Remote. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:89: DB::Exception::Exception(String const&, int, bool) @ 0x000000000a9434aa
5. /build/src/Common/Exception.h:97: DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000a943455
6. /build/src/IO/ReadHelpers.cpp:1560: DB::readException(DB::ReadBuffer&, String const&, bool) @ 0x000000001336a782
7. /build/src/Client/Connection.cpp:1233: DB::Connection::receiveException() const @ 0x000000001d2be459
8. /build/src/Client/Connection.cpp:1070: DB::Connection::receivePacket() @ 0x000000001d2c59b1
9. /build/src/Client/PacketReceiver.cpp:66: DB::PacketReceiver::Task::run(std::function<void (int, Poco::Timespan, DB::AsyncEventTimeoutType, String const&, unsigned int)>, std::function<void ()>) @ 0x000000001d3328a4
10. /build/src/Common/AsyncTaskExecutor.cpp:88: DB::AsyncTaskExecutor::Routine::operator()(std::function<void ()>) @ 0x0000000019ebe669
11. /build/src/Common/Fiber.h:76: Fiber::RoutineImpl<DB::AsyncTaskExecutor::Routine>::operator()(boost::context::fiber&&) @ 0x0000000019ebe578
12. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x0000000019ebe508
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:531: ? @ 0x0000000019ebe4c8
14. /build/contrib/boost/boost/context/fiber_fcontext.hpp:157: boost::context::detail::fiber_record<boost::context::fiber, FiberStack&, Fiber::RoutineImpl<DB::AsyncTaskExecutor::Routine>>::run(void*) @ 0x0000000019ebe3fe
15. /build/contrib/boost/boost/context/fiber_fcontext.hpp:97: void boost::context::detail::fiber_entry<boost::context::detail::fiber_record<boost::context::fiber, FiberStack&, Fiber::RoutineImpl<DB::AsyncTaskExecutor::Routine>>>(boost::context::detail::transfer_t) @ 0x0000000019ebe11d

2024.11.05 19:41:10.544661 [ 4766 ] {27755a27-0b2d-47c9-8f18-d7498bcfd697} <Error> executeQuery: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:49888). (NETWORK_ERROR) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49888) (comment: 00012_sorting_distributed.sql) (in query: SELECT CAST(`__table1`.`EventTime`, 'DateTime(\'Asia/Dubai\')') AS `CAST(EventTime, 'DateTime(\\'Asia/Dubai\\')')` FROM `test`.`hits` AS `__table1` ORDER BY `__table1`.`EventTime` DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:158: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e8fa4
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Server/TCPHandler.cpp:2217: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001d47db8f
9. /build/src/Server/TCPHandler.cpp:1084: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b9aa
10. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:41:12.120795 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/wxs/ohvagbxgrrllfuyejjfvfxcqkjrvi, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 01a1527a-abb6-4cfd-9ace-152ea0890c55, upload has finished false.
2024.11.05 19:41:12.127584 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/wxs/ohvagbxgrrllfuyejjfvfxcqkjrvi, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 01a1527a-abb6-4cfd-9ace-152ea0890c55, upload has finished false
2024.11.05 19:41:12.205200 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/wxs/ohvagbxgrrllfuyejjfvfxcqkjrvi, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 01a1527a-abb6-4cfd-9ace-152ea0890c55, upload has finished false
2024.11.05 19:41:12.237228 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 13 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:13.952750 [ 4766 ] {27755a27-0b2d-47c9-8f18-d7498bcfd697} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:49888). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:169: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e9244
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Server/TCPHandler.cpp:2253: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001d487855
9. /build/src/Server/TCPHandler.cpp:2330: DB::TCPHandler::sendLogs() @ 0x000000001d47c5ff
10. /build/src/Server/TCPHandler.cpp:693: DB::TCPHandler::runImpl() @ 0x000000001d471ac2
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:13.962138 [ 4766 ] {27755a27-0b2d-47c9-8f18-d7498bcfd697} <Error> TCPHandler: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:49888). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:158: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e8fa4
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Server/TCPHandler.cpp:2217: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001d47db8f
9. /build/src/Server/TCPHandler.cpp:1084: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b9aa
10. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:41:14.005803 [ 4766 ] {27755a27-0b2d-47c9-8f18-d7498bcfd697} <Warning> TCPHandler: Client has gone away.
2024.11.05 19:41:14.213768 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:19.292758 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/gja/luogkozryfmttwvjhrrdxicyhfjan, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 41adf44b-de79-4ff1-8645-e6528fbea6ca, upload has finished false.
2024.11.05 19:41:19.300961 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/gja/luogkozryfmttwvjhrrdxicyhfjan, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 41adf44b-de79-4ff1-8645-e6528fbea6ca, upload has finished false
2024.11.05 19:41:19.327550 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/gja/luogkozryfmttwvjhrrdxicyhfjan, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 41adf44b-de79-4ff1-8645-e6528fbea6ca, upload has finished false
2024.11.05 19:41:19.453905 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 13 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:21.578948 [ 3568 ] {} <Warning> CgroupsMemoryUsageObserver: Exceeded soft memory limit (13.77 GiB)
2024.11.05 19:41:24.717338 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 8 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:27.115737 [ 3800 ] {aba1143f-35fc-44d5-9ac6-92d1321bd209} <Error> executeQuery: Code: 60. DB::Exception: Table test_3.unsigned_integer_test_table does not exist. Maybe you meant test_3.signed_integer_test_table?. (UNKNOWN_TABLE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:50844) (comment: 00653_verification_monotonic_data_load.sh) (in query: INSERT INTO unsigned_integer_test_table VALUES ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&, String&&) @ 0x000000000aa272e4
6. /build/src/Databases/IDatabase.cpp:39: DB::IDatabase::getTable(String const&, std::shared_ptr<DB::Context const>) const @ 0x000000001a1465cb
7. /build/src/Interpreters/DatabaseCatalog.cpp:429: DB::DatabaseCatalog::getTableImpl(DB::StorageID const&, std::shared_ptr<DB::Context const>, std::optional<DB::Exception>*) const @ 0x000000001a670796
8. /build/src/Interpreters/DatabaseCatalog.cpp:952: DB::DatabaseCatalog::getTable(DB::StorageID const&, std::shared_ptr<DB::Context const>) const @ 0x000000001a675ae3
9. /build/src/Interpreters/InterpreterInsertQuery.cpp:123: DB::InterpreterInsertQuery::getTable(DB::ASTInsertQuery&) @ 0x000000001b429f12
10. /build/src/Interpreters/InterpreterInsertQuery.cpp:406: DB::InterpreterInsertQuery::execute() @ 0x000000001b42c384
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:41:27.123045 [ 3800 ] {aba1143f-35fc-44d5-9ac6-92d1321bd209} <Error> TCPHandler: Code: 60. DB::Exception: Table test_3.unsigned_integer_test_table does not exist. Maybe you meant test_3.signed_integer_test_table?. (UNKNOWN_TABLE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&, String&&) @ 0x000000000aa272e4
6. /build/src/Databases/IDatabase.cpp:39: DB::IDatabase::getTable(String const&, std::shared_ptr<DB::Context const>) const @ 0x000000001a1465cb
7. /build/src/Interpreters/DatabaseCatalog.cpp:429: DB::DatabaseCatalog::getTableImpl(DB::StorageID const&, std::shared_ptr<DB::Context const>, std::optional<DB::Exception>*) const @ 0x000000001a670796
8. /build/src/Interpreters/DatabaseCatalog.cpp:952: DB::DatabaseCatalog::getTable(DB::StorageID const&, std::shared_ptr<DB::Context const>) const @ 0x000000001a675ae3
9. /build/src/Interpreters/InterpreterInsertQuery.cpp:123: DB::InterpreterInsertQuery::getTable(DB::ASTInsertQuery&) @ 0x000000001b429f12
10. /build/src/Interpreters/InterpreterInsertQuery.cpp:406: DB::InterpreterInsertQuery::execute() @ 0x000000001b42c384
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:41:30.764772 [ 2814 ] {5b524e1b-bf0b-4217-8416-0edb0a783ecd} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:50852) (comment: 00653_verification_monotonic_data_load.sh) (in query: INSERT INTO enum_test_table VALUES ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a9c
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d

2024.11.05 19:41:30.778994 [ 2814 ] {5b524e1b-bf0b-4217-8416-0edb0a783ecd} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a9c
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d

2024.11.05 19:41:31.004136 [ 3641 ] {} <Error> void DB::SystemLog<DB::PartLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::PartLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:31.495333 [ 3800 ] {a1d11d37-41ce-4c6c-80a9-a04d4a489826} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=3): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:50848) (comment: 00159_parallel_formatting_json_and_friends_1.sh) (in query: SELECT ClientEventTime::DateTime('Asia/Dubai') as a, MobilePhoneModel as b, ClientIP6 as c FROM test.hits ORDER BY a, b, c LIMIT 3000000 Format JSONCompactEachRowWithNames), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:41:31.710379 [ 3800 ] {a1d11d37-41ce-4c6c-80a9-a04d4a489826} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=3): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:41:32.047307 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/dlb/djtfrctcttmrtrdsnoeilfarcxntr, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0e93633d-9fe9-4ac1-a43f-e5d52e36e172, upload has finished false.
2024.11.05 19:41:32.048795 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/dlb/djtfrctcttmrtrdsnoeilfarcxntr, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0e93633d-9fe9-4ac1-a43f-e5d52e36e172, upload has finished false
2024.11.05 19:41:32.141697 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/dlb/djtfrctcttmrtrdsnoeilfarcxntr, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0e93633d-9fe9-4ac1-a43f-e5d52e36e172, upload has finished false
2024.11.05 19:41:32.176835 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 22 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:36.714684 [ 2814 ] {e46e0f9b-88e9-4b8e-9e14-fbbec1a9e692} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=11, jobs=8). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:50862) (comment: 00171_grouping_aggregated_transform_bug.sql) (in query: SELECT sum(cityHash64(*)) FROM (SELECT CounterID, quantileTiming(0.5)(SendTiming), count() FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9,10}', test.hits) WHERE SendTiming != -1 GROUP BY CounterID) SETTINGS max_block_size = 63169;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:41:37.312019 [ 2814 ] {e46e0f9b-88e9-4b8e-9e14-fbbec1a9e692} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=11, jobs=8). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:41:38.709239 [ 2816 ] {f5e0e24a-3d51-48c4-b501-d11cb83886c6} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=31, jobs=31). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:58568) (comment: 00002_count_visits.sql) (in query: SELECT sum(Sign) FROM test.visits ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:41:39.315417 [ 2816 ] {f5e0e24a-3d51-48c4-b501-d11cb83886c6} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=31, jobs=31). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:41:43.647397 [ 3612 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Removing temporary directory /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:41:43.648023 [ 3612 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:41:43.671620 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/crk/eitgihqhtrnxefovalohmpvyqvvyq, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f894f853-5952-4c20-853e-23742e115345, upload has finished false.
2024.11.05 19:41:43.673584 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/crk/eitgihqhtrnxefovalohmpvyqvvyq, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f894f853-5952-4c20-853e-23742e115345, upload has finished false
2024.11.05 19:41:43.724487 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/crk/eitgihqhtrnxefovalohmpvyqvvyq, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f894f853-5952-4c20-853e-23742e115345, upload has finished false
2024.11.05 19:41:44.104085 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Error> virtual void DB::MetadataStorageFromDiskTransaction::commit(): Code: 521. DB::ErrnoException: Paths cannot be exchanged because /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1 or /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/delete_tmp_tmp_merge_all_1_5_1 does not exist: , errno: 2, strerror: No such file or directory. (ATOMIC_RENAME_FAIL), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:194: DB::ErrnoException::ErrnoException<String const&, String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String const&>::type>, String const&, String const&) @ 0x000000001a0d58cc
6. /build/src/Common/atomicRename.cpp:93: DB::renameat2(String const&, String const&, int) @ 0x000000001a0d53c3
7. /build/src/Common/atomicRename.cpp:211: DB::renameNoReplace(String const&, String const&) @ 0x000000001a0d5102
8. /build/src/Disks/DiskLocal.cpp:320: DB::DiskLocal::moveFile(String const&, String const&) @ 0x000000001a2d53f9
9. /build/src/Disks/ObjectStorages/MetadataStorageFromDiskTransactionOperations.cpp:207: DB::MoveFileOperation::execute(std::unique_lock<DB::SharedMutex>&) @ 0x000000001a3af5cc
10. /build/src/Disks/ObjectStorages/MetadataStorageFromDisk.cpp:186: DB::MetadataStorageFromDiskTransaction::commit() @ 0x000000001a3a369a
11. /build/src/Disks/ObjectStorages/DiskObjectStorageTransaction.cpp:925: DB::DiskObjectStorageTransaction::commit() @ 0x000000001a340c97
12. /build/src/Disks/ObjectStorages/DiskObjectStorage.cpp:197: DB::DiskObjectStorage::moveFile(String const&, String const&, bool) @ 0x000000001a319589
13. /build/src/Disks/ObjectStorages/DiskObjectStorage.cpp:227: DB::DiskObjectStorage::moveFile(String const&, String const&) @ 0x000000001a319a50
14. /build/src/Disks/ObjectStorages/DiskObjectStorage.h:122: DB::DiskObjectStorage::moveDirectory(String const&, String const&) @ 0x000000001a3231a9
15. /build/src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp:666: DB::DataPartStorageOnDiskBase::remove(std::function<DB::CanRemoveDescription ()>&&, DB::MergeTreeDataPartChecksums const&, std::list<DB::IDataPartStorage::ProjectionChecksums, std::allocator<DB::IDataPartStorage::ProjectionChecksums>>, bool, std::shared_ptr<Poco::Logger>) @ 0x000000001ca91998
16. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:1959: DB::IMergeTreeDataPart::remove() @ 0x000000001cab4ce8
17. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:548: DB::IMergeTreeDataPart::removeIfNeeded() @ 0x000000001cab32c7
18. /build/src/Storages/MergeTree/MergeTreeDataPartWide.cpp:189: DB::MergeTreeDataPartWide::~MergeTreeDataPartWide() @ 0x000000001cccc463
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:64: void std::__destroy_at[abi:v15000]<DB::MergeTreeDataPartWide, 0>(DB::MergeTreeDataPartWide*) @ 0x000000001ccb508e
20. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:89: void std::destroy_at[abi:v15000]<DB::MergeTreeDataPartWide, 0>(DB::MergeTreeDataPartWide*) @ 0x000000001ccb5035
21. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:321: void std::allocator_traits<std::allocator<DB::MergeTreeDataPartWide>>::destroy[abi:v15000]<DB::MergeTreeDataPartWide, void, void>(std::allocator<DB::MergeTreeDataPartWide>&, DB::MergeTreeDataPartWide*) @ 0x000000001ccb5019
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:309: std::__shared_ptr_emplace<DB::MergeTreeDataPartWide, std::allocator<DB::MergeTreeDataPartWide>>::__on_zero_shared() @ 0x000000001ccb4e6e
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:175: std::__shared_count::__release_shared[abi:v15000]() @ 0x000000000a8f6c14
24. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215: std::__shared_weak_count::__release_shared[abi:v15000]() @ 0x000000000a8f6bb9
25. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:703: std::shared_ptr<DB::IMergeTreeDataPart>::~shared_ptr[abi:v15000]() @ 0x000000001ba2ab0c
26. /build/src/Storages/MergeTree/IMergeTreeDataPartWriter.cpp:63: DB::IMergeTreeDataPartWriter::~IMergeTreeDataPartWriter() @ 0x000000001ccdf932
27. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.h:43: DB::MergeTreeDataPartWriterOnDisk::~MergeTreeDataPartWriterOnDisk() @ 0x000000001ccd6037
28. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.h:20: DB::MergeTreeDataPartWriterWide::~MergeTreeDataPartWriterWide() @ 0x000000001ccf8d37
29. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.h:20: DB::MergeTreeDataPartWriterWide::~MergeTreeDataPartWriterWide() @ 0x000000001ccf8d59
30. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:49: std::default_delete<DB::IMergeTreeDataPartWriter>::operator()[abi:v15000](DB::IMergeTreeDataPartWriter*) const @ 0x000000001cb7ccf4
31. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:306: std::unique_ptr<DB::IMergeTreeDataPartWriter, std::default_delete<DB::IMergeTreeDataPartWriter>>::reset[abi:v15000](DB::IMergeTreeDataPartWriter*) @ 0x000000001cb7cc5c
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:44.375244 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:45: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce610d
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
23. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
24. /build/src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp:69: DB::MergedColumnOnlyOutputStream::fillChecksums(std::shared_ptr<DB::IMergeTreeDataPart>&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3fe1a
25. /build/src/Storages/MergeTree/MergeTask.cpp:657: DB::MergeTask::VerticalMergeStage::finalizeVerticalMergeForOneColumn() const @ 0x000000001cb6fabd
26. /build/src/Storages/MergeTree/MergeTask.cpp:877: DB::MergeTask::VerticalMergeStage::executeVerticalMergeForAllColumns() const @ 0x000000001cb72f6e
27. /build/src/Storages/MergeTree/MergeTask.cpp:829: DB::MergeTask::VerticalMergeStage::execute() @ 0x000000001cb72cb8
28. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
29. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:44.385750 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Error> void DB::IMergeTreeDataPart::removeIfNeeded(): while removing part all_1_5_1 with path store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/: Code: 521. DB::ErrnoException: Paths cannot be exchanged because /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/delete_tmp_tmp_merge_all_1_5_1 or /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1 does not exist: , errno: 2, strerror: No such file or directory: While rolling back operation #0. (ATOMIC_RENAME_FAIL), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:194: DB::ErrnoException::ErrnoException<String const&, String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String const&>::type>, String const&, String const&) @ 0x000000001a0d58cc
6. /build/src/Common/atomicRename.cpp:93: DB::renameat2(String const&, String const&, int) @ 0x000000001a0d53c3
7. /build/src/Common/atomicRename.cpp:211: DB::renameNoReplace(String const&, String const&) @ 0x000000001a0d5102
8. /build/src/Disks/DiskLocal.cpp:320: DB::DiskLocal::moveFile(String const&, String const&) @ 0x000000001a2d53f9
9. /build/src/Disks/ObjectStorages/MetadataStorageFromDiskTransactionOperations.cpp:212: DB::MoveFileOperation::undo() @ 0x000000001a3af608
10. /build/src/Disks/ObjectStorages/MetadataStorageFromDisk.cpp:224: DB::MetadataStorageFromDiskTransaction::rollback(unsigned long) @ 0x000000001a3a3c78
11. /build/src/Disks/ObjectStorages/MetadataStorageFromDisk.cpp:193: DB::MetadataStorageFromDiskTransaction::commit() @ 0x000000001a3a38ab
12. /build/src/Disks/ObjectStorages/DiskObjectStorageTransaction.cpp:925: DB::DiskObjectStorageTransaction::commit() @ 0x000000001a340c97
13. /build/src/Disks/ObjectStorages/DiskObjectStorage.cpp:197: DB::DiskObjectStorage::moveFile(String const&, String const&, bool) @ 0x000000001a319589
14. /build/src/Disks/ObjectStorages/DiskObjectStorage.cpp:227: DB::DiskObjectStorage::moveFile(String const&, String const&) @ 0x000000001a319a50
15. /build/src/Disks/ObjectStorages/DiskObjectStorage.h:122: DB::DiskObjectStorage::moveDirectory(String const&, String const&) @ 0x000000001a3231a9
16. /build/src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp:666: DB::DataPartStorageOnDiskBase::remove(std::function<DB::CanRemoveDescription ()>&&, DB::MergeTreeDataPartChecksums const&, std::list<DB::IDataPartStorage::ProjectionChecksums, std::allocator<DB::IDataPartStorage::ProjectionChecksums>>, bool, std::shared_ptr<Poco::Logger>) @ 0x000000001ca91998
17. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:1959: DB::IMergeTreeDataPart::remove() @ 0x000000001cab4ce8
18. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:548: DB::IMergeTreeDataPart::removeIfNeeded() @ 0x000000001cab32c7
19. /build/src/Storages/MergeTree/MergeTreeDataPartWide.cpp:189: DB::MergeTreeDataPartWide::~MergeTreeDataPartWide() @ 0x000000001cccc463
20. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:64: void std::__destroy_at[abi:v15000]<DB::MergeTreeDataPartWide, 0>(DB::MergeTreeDataPartWide*) @ 0x000000001ccb508e
21. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:89: void std::destroy_at[abi:v15000]<DB::MergeTreeDataPartWide, 0>(DB::MergeTreeDataPartWide*) @ 0x000000001ccb5035
22. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:321: void std::allocator_traits<std::allocator<DB::MergeTreeDataPartWide>>::destroy[abi:v15000]<DB::MergeTreeDataPartWide, void, void>(std::allocator<DB::MergeTreeDataPartWide>&, DB::MergeTreeDataPartWide*) @ 0x000000001ccb5019
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:309: std::__shared_ptr_emplace<DB::MergeTreeDataPartWide, std::allocator<DB::MergeTreeDataPartWide>>::__on_zero_shared() @ 0x000000001ccb4e6e
24. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:175: std::__shared_count::__release_shared[abi:v15000]() @ 0x000000000a8f6c14
25. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215: std::__shared_weak_count::__release_shared[abi:v15000]() @ 0x000000000a8f6bb9
26. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:703: std::shared_ptr<DB::IMergeTreeDataPart>::~shared_ptr[abi:v15000]() @ 0x000000001ba2ab0c
27. /build/src/Storages/MergeTree/IMergeTreeDataPartWriter.cpp:63: DB::IMergeTreeDataPartWriter::~IMergeTreeDataPartWriter() @ 0x000000001ccdf932
28. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.h:43: DB::MergeTreeDataPartWriterOnDisk::~MergeTreeDataPartWriterOnDisk() @ 0x000000001ccd6037
29. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.h:20: DB::MergeTreeDataPartWriterWide::~MergeTreeDataPartWriterWide() @ 0x000000001ccf8d37
30. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.h:20: DB::MergeTreeDataPartWriterWide::~MergeTreeDataPartWriterWide() @ 0x000000001ccf8d59
31. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:49: std::default_delete<DB::IMergeTreeDataPartWriter>::operator()[abi:v15000](DB::IMergeTreeDataPartWriter*) const @ 0x000000001cb7ccf4
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:44.448797 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 33 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:47.509267 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:47.529688 [ 2814 ] {5e2f8412-0dc1-403c-8724-5056bccaf079} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=19, jobs=19). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48428) (comment: 00006_agregates.sql) (in query: SELECT sum(AdvEngineID), count(), avg(ResolutionWidth) FROM test.hits ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:41:47.935309 [ 2816 ] {b2c79466-0872-4e46-b7e6-b00e2261b5ee} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=11, jobs=11). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48432) (comment: 00653_verification_monotonic_data_load.sh) (in query: SELECT count() FROM signed_integer_test_table WHERE toUInt64(val) == 0;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:41:47.970608 [ 2816 ] {b2c79466-0872-4e46-b7e6-b00e2261b5ee} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=11, jobs=11). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:41:48.675887 [ 2814 ] {5e2f8412-0dc1-403c-8724-5056bccaf079} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=19, jobs=19). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:41:51.164811 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column VisitID): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_5_5_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 0 with max_rows_to_read = 6739): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:51.545755 [ 2816 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::1]:48434). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x00000000134e772c
6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x00000000134e68a0
7. /build/src/IO/ReadBuffer.h:70: DB::ReadBuffer::next() @ 0x000000000af5f46a
8. /build/src/IO/ReadBuffer.h:106: DB::TCPHandler::runImpl() @ 0x000000001d46f277
9. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
15. ? @ 0x00007f3c91188ac3
16. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:54.447945 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column StartDate): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_1_1_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 0 with max_rows_to_read = 6658): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:55.661105 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:55.946226 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/bgp/lamjxymmjdciunpwtjlekkurxamrx, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b04a22e0-11ac-4986-a324-fba25f318949, upload has finished false.
2024.11.05 19:41:55.946976 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/bgp/lamjxymmjdciunpwtjlekkurxamrx, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b04a22e0-11ac-4986-a324-fba25f318949, upload has finished false
2024.11.05 19:41:56.006297 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/bgp/lamjxymmjdciunpwtjlekkurxamrx, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b04a22e0-11ac-4986-a324-fba25f318949, upload has finished false
2024.11.05 19:41:56.046916 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 32 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:41:57.478685 [ 2816 ] {a91f56f6-0a41-4050-9e03-490eb9d729be} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60514) (comment: 00653_verification_monotonic_data_load.sh) (in query: SELECT count() FROM signed_integer_test_table WHERE toInt16(val) == 0;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:41:57.520085 [ 2816 ] {a91f56f6-0a41-4050-9e03-490eb9d729be} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:41:59.657558 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=29): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 4 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:00.247240 [ 4766 ] {2b17c1a0-0970-4f05-be88-76f57ae94844} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60498) (comment: 00071_merge_tree_optimize_aio.sql) (in query: INSERT INTO test.hits_snippet(EventTime, EventDate, CounterID, UserID, URL, Referer) SELECT EventTime, EventDate, CounterID, UserID, URL, Referer FROM test.hits WHERE EventDate = toDate('2014-03-18') ORDER BY EventTime, EventDate, CounterID, UserID, URL, Referer ASC LIMIT 50;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:42:00.747065 [ 2816 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::1]:60522). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x00000000134e772c
6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x00000000134e68a0
7. /build/src/IO/ReadBuffer.h:70: DB::ReadBuffer::next() @ 0x000000000af5f46a
8. /build/src/IO/ReadBuffer.h:106: DB::TCPHandler::runImpl() @ 0x000000001d46f277
9. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
15. ? @ 0x00007f3c91188ac3
16. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:03.298688 [ 4766 ] {2b17c1a0-0970-4f05-be88-76f57ae94844} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:42:03.764400 [ 4767 ] {7dda192e-c1d2-4f3e-80af-7fb4b8432a1f} <Error> executeQuery: Code: 60. DB::Exception: Unknown table expression identifier 'unsigned_integer_test_table' in scope SELECT count() FROM unsigned_integer_test_table WHERE toInt64(val) = 0. (UNKNOWN_TABLE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60248) (comment: 00653_verification_monotonic_data_load.sh) (in query: SELECT count() FROM unsigned_integer_test_table WHERE toInt64(val) == 0;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6970: DB::(anonymous namespace)::QueryAnalyzer::initializeQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0aa552
7. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8159: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4a3
8. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
9. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
10. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
13. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
14. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
21. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
22. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
23. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
24. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
25. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
26. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
27. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
28. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
29. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
30. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
31. ? @ 0x00007f3c91188ac3

2024.11.05 19:42:03.850708 [ 4767 ] {7dda192e-c1d2-4f3e-80af-7fb4b8432a1f} <Error> TCPHandler: Code: 60. DB::Exception: Unknown table expression identifier 'unsigned_integer_test_table' in scope SELECT count() FROM unsigned_integer_test_table WHERE toInt64(val) = 0. (UNKNOWN_TABLE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6970: DB::(anonymous namespace)::QueryAnalyzer::initializeQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0aa552
7. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8159: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4a3
8. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
9. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
10. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
13. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
14. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
21. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
22. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
23. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
24. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
25. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
26. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
27. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
28. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
29. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
30. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
31. ? @ 0x00007f3c91188ac3

2024.11.05 19:42:06.532793 [ 2814 ] {81246be3-a84f-4fc6-845c-ed978e89c48a} <Error> executeQuery: Code: 60. DB::Exception: Unknown table expression identifier 'unsigned_integer_test_table' in scope SELECT count() FROM unsigned_integer_test_table WHERE toUInt64(val) = 0. (UNKNOWN_TABLE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60262) (comment: 00653_verification_monotonic_data_load.sh) (in query: SELECT count() FROM unsigned_integer_test_table WHERE toUInt64(val) == 0;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6970: DB::(anonymous namespace)::QueryAnalyzer::initializeQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0aa552
7. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8159: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4a3
8. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
9. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
10. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
13. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
14. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
21. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
22. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
23. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
24. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
25. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
26. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
27. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
28. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
29. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
30. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
31. ? @ 0x00007f3c91188ac3

2024.11.05 19:42:06.556336 [ 2814 ] {81246be3-a84f-4fc6-845c-ed978e89c48a} <Error> TCPHandler: Code: 60. DB::Exception: Unknown table expression identifier 'unsigned_integer_test_table' in scope SELECT count() FROM unsigned_integer_test_table WHERE toUInt64(val) = 0. (UNKNOWN_TABLE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6970: DB::(anonymous namespace)::QueryAnalyzer::initializeQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0aa552
7. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8159: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4a3
8. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
9. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
10. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
13. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
14. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
21. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
22. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
23. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
24. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
25. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
26. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
27. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
28. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
29. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
30. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
31. ? @ 0x00007f3c91188ac3

2024.11.05 19:42:06.576270 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:07.430945 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/eym/zmgqgxxjoimpvozsxbatzyybccxsp, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 9198146d-d703-4398-b43e-43767c35e009, upload has finished false.
2024.11.05 19:42:07.434032 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/eym/zmgqgxxjoimpvozsxbatzyybccxsp, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 9198146d-d703-4398-b43e-43767c35e009, upload has finished false
2024.11.05 19:42:07.486400 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/eym/zmgqgxxjoimpvozsxbatzyybccxsp, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 9198146d-d703-4398-b43e-43767c35e009, upload has finished false
2024.11.05 19:42:07.577013 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 20 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:09.411476 [ 4767 ] {fb542e20-b777-4926-bdc0-7d2e2bac07ed} <Error> executeQuery: Code: 60. DB::Exception: Unknown table expression identifier 'unsigned_integer_test_table' in scope SELECT count() FROM unsigned_integer_test_table WHERE toInt32(val) = 0. (UNKNOWN_TABLE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60276) (comment: 00653_verification_monotonic_data_load.sh) (in query: SELECT count() FROM unsigned_integer_test_table WHERE toInt32(val) == 0;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6970: DB::(anonymous namespace)::QueryAnalyzer::initializeQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0aa552
7. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8159: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4a3
8. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
9. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
10. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
13. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
14. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
21. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
22. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
23. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
24. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
25. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
26. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
27. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
28. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
29. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
30. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
31. ? @ 0x00007f3c91188ac3

2024.11.05 19:42:09.432936 [ 4767 ] {fb542e20-b777-4926-bdc0-7d2e2bac07ed} <Error> TCPHandler: Code: 60. DB::Exception: Unknown table expression identifier 'unsigned_integer_test_table' in scope SELECT count() FROM unsigned_integer_test_table WHERE toInt32(val) = 0. (UNKNOWN_TABLE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6970: DB::(anonymous namespace)::QueryAnalyzer::initializeQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0aa552
7. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8159: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4a3
8. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
9. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
10. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
13. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
14. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
21. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
22. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
23. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
24. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
25. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
26. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
27. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
28. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
29. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
30. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
31. ? @ 0x00007f3c91188ac3

2024.11.05 19:42:12.178605 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:12.747698 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:12.765288 [ 4767 ] {e499b362-e418-4e42-a7ce-fe8d32bfdf7c} <Error> executeQuery: Code: 60. DB::Exception: Unknown table expression identifier 'unsigned_integer_test_table' in scope SELECT count() FROM unsigned_integer_test_table WHERE toUInt32(val) = 0. (UNKNOWN_TABLE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:43674) (comment: 00653_verification_monotonic_data_load.sh) (in query: SELECT count() FROM unsigned_integer_test_table WHERE toUInt32(val) == 0;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6970: DB::(anonymous namespace)::QueryAnalyzer::initializeQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0aa552
7. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8159: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4a3
8. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
9. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
10. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
13. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
14. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
21. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
22. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
23. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
24. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
25. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
26. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
27. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
28. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
29. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
30. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
31. ? @ 0x00007f3c91188ac3

2024.11.05 19:42:12.795477 [ 4767 ] {e499b362-e418-4e42-a7ce-fe8d32bfdf7c} <Error> TCPHandler: Code: 60. DB::Exception: Unknown table expression identifier 'unsigned_integer_test_table' in scope SELECT count() FROM unsigned_integer_test_table WHERE toUInt32(val) = 0. (UNKNOWN_TABLE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6970: DB::(anonymous namespace)::QueryAnalyzer::initializeQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0aa552
7. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8159: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4a3
8. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
9. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
10. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
13. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
14. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
21. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
22. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
23. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
24. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
25. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
26. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
27. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
28. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
29. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
30. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
31. ? @ 0x00007f3c91188ac3

2024.11.05 19:42:12.846737 [ 2814 ] {348a4d94-c90b-4767-bf81-2e9c1f9de64d} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=8): (while reading column SearchEngineID): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/5e7/5e75bc07-5cbc-4ba8-9da3-6eb03e2db7cd/201403_7_13_1/ in table test.hits (5e75bc07-5cbc-4ba8-9da3-6eb03e2db7cd) located on disk s3_disk of type s3, from mark 96 with max_rows_to_read = 93848): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60270) (comment: 00036_aggregate_hashed.sql) (in query: SELECT SearchEngineID AS k1, SearchPhrase AS k2, count() AS c FROM test.hits GROUP BY k1, k2 ORDER BY c DESC, k1, k2 LIMIT 10 ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110

2024.11.05 19:42:15.091903 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/udc/htxtcbbmkyrnfejgcwddjiuwrmrre, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0edb3e31-a5ae-41d4-a3e3-9916046a4039, upload has finished false.
2024.11.05 19:42:15.093149 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/udc/htxtcbbmkyrnfejgcwddjiuwrmrre, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0edb3e31-a5ae-41d4-a3e3-9916046a4039, upload has finished false
2024.11.05 19:42:15.148692 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/udc/htxtcbbmkyrnfejgcwddjiuwrmrre, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0edb3e31-a5ae-41d4-a3e3-9916046a4039, upload has finished false
2024.11.05 19:42:15.368841 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 18 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:15.418388 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:15.714573 [ 4767 ] {e9bebe25-e293-4021-afc9-02523c899da3} <Error> executeQuery: Code: 60. DB::Exception: Unknown table expression identifier 'unsigned_integer_test_table' in scope SELECT count() FROM unsigned_integer_test_table WHERE toInt16(val) = 0. (UNKNOWN_TABLE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:43682) (comment: 00653_verification_monotonic_data_load.sh) (in query: SELECT count() FROM unsigned_integer_test_table WHERE toInt16(val) == 0;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6970: DB::(anonymous namespace)::QueryAnalyzer::initializeQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0aa552
7. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8159: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4a3
8. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
9. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
10. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
13. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
14. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
21. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
22. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
23. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
24. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
25. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
26. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
27. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
28. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
29. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
30. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
31. ? @ 0x00007f3c91188ac3

2024.11.05 19:42:15.729639 [ 4767 ] {e9bebe25-e293-4021-afc9-02523c899da3} <Error> TCPHandler: Code: 60. DB::Exception: Unknown table expression identifier 'unsigned_integer_test_table' in scope SELECT count() FROM unsigned_integer_test_table WHERE toInt16(val) = 0. (UNKNOWN_TABLE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6970: DB::(anonymous namespace)::QueryAnalyzer::initializeQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0aa552
7. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8159: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4a3
8. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
9. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
10. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
13. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
14. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
21. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
22. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
23. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
24. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
25. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
26. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
27. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
28. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
29. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
30. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
31. ? @ 0x00007f3c91188ac3

2024.11.05 19:42:17.897791 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 2 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:17.927286 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=1): (while reading column VisitID): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_4_4_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 19 with max_rows_to_read = 6741): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:18.591220 [ 4767 ] {54cae60d-b12b-48c1-bb76-c59b68f50fdb} <Error> executeQuery: Code: 60. DB::Exception: Unknown table expression identifier 'unsigned_integer_test_table' in scope SELECT count() FROM unsigned_integer_test_table WHERE toUInt16(val) = 0. (UNKNOWN_TABLE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:43686) (comment: 00653_verification_monotonic_data_load.sh) (in query: SELECT count() FROM unsigned_integer_test_table WHERE toUInt16(val) == 0;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6970: DB::(anonymous namespace)::QueryAnalyzer::initializeQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0aa552
7. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8159: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4a3
8. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
9. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
10. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
13. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
14. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
21. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
22. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
23. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
24. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
25. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
26. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
27. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
28. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
29. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
30. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
31. ? @ 0x00007f3c91188ac3

2024.11.05 19:42:18.615324 [ 4767 ] {54cae60d-b12b-48c1-bb76-c59b68f50fdb} <Error> TCPHandler: Code: 60. DB::Exception: Unknown table expression identifier 'unsigned_integer_test_table' in scope SELECT count() FROM unsigned_integer_test_table WHERE toUInt16(val) = 0. (UNKNOWN_TABLE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6970: DB::(anonymous namespace)::QueryAnalyzer::initializeQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0aa552
7. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8159: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4a3
8. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
9. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
10. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
12. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
13. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
14. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
21. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
22. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
23. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
24. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
25. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
26. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
27. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
28. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
29. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
30. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
31. ? @ 0x00007f3c91188ac3

2024.11.05 19:42:19.357590 [ 2814 ] {348a4d94-c90b-4767-bf81-2e9c1f9de64d} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=8): (while reading column SearchEngineID): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/5e7/5e75bc07-5cbc-4ba8-9da3-6eb03e2db7cd/201403_7_13_1/ in table test.hits (5e75bc07-5cbc-4ba8-9da3-6eb03e2db7cd) located on disk s3_disk of type s3, from mark 96 with max_rows_to_read = 93848): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110

2024.11.05 19:42:22.227115 [ 2814 ] {04cdcd27-1a64-4f3f-9b73-a597a281fa23} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=10, jobs=10). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:43688) (comment: 00653_verification_monotonic_data_load.sh) (in query: SELECT count() FROM enum_test_table WHERE toInt32(val) == 1;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:42:22.241318 [ 2814 ] {04cdcd27-1a64-4f3f-9b73-a597a281fa23} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=10, jobs=10). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:42:22.756022 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 8 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:27.970697 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 11 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:28.762013 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:35.771878 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:40.452664 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/kqi/fqvzhujwbvqeoprenlxemfebrxdak, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 36e427ac-1910-4eb6-8b67-4e9703030238, upload has finished false.
2024.11.05 19:42:40.453694 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/kqi/fqvzhujwbvqeoprenlxemfebrxdak, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 36e427ac-1910-4eb6-8b67-4e9703030238, upload has finished false
2024.11.05 19:42:40.515212 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/kqi/fqvzhujwbvqeoprenlxemfebrxdak, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 36e427ac-1910-4eb6-8b67-4e9703030238, upload has finished false
2024.11.05 19:42:40.554455 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 35 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:41.139036 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {62ec4990-945b-4d8e-9889-ca33c67862f8::202411_1_31_6}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
25. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
26. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
27. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
28. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:43.150884 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0e641ae6-a90e-4cec-8ebd-a21f62e7934a::202411_1_26_5}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:43.388874 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0e641ae6-a90e-4cec-8ebd-a21f62e7934a::202411_1_26_5}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:44.762986 [ 4767 ] {d7a9326e-5528-44f5-bda4-a029771e0759} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:36716) (comment: 00166_explain_estimate.sql) (in query: EXPLAIN ESTIMATE SELECT count() FROM test.hits WHERE CounterID = 29103473 UNION ALL SELECT count() FROM test.hits WHERE CounterID = 1704509;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterExplainQuery.cpp:560: DB::InterpreterExplainQuery::executeImpl() @ 0x000000001b043c5f
18. /build/src/Interpreters/InterpreterExplainQuery.cpp:91: DB::InterpreterExplainQuery::execute() @ 0x000000001b04162b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:42:44.793853 [ 4767 ] {d7a9326e-5528-44f5-bda4-a029771e0759} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterExplainQuery.cpp:560: DB::InterpreterExplainQuery::executeImpl() @ 0x000000001b043c5f
18. /build/src/Interpreters/InterpreterExplainQuery.cpp:91: DB::InterpreterExplainQuery::execute() @ 0x000000001b04162b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:42:45.775081 [ 3641 ] {} <Error> void DB::SystemLog<DB::PartLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::PartLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:47.140363 [ 4766 ] {20252ba7-a59e-4b19-a3a7-a3b2afdd4421} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:38368) (comment: 00653_verification_monotonic_data_load.sh) (in query: SELECT count() FROM date_test_table WHERE toInt16(val) == 1;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:42:47.187613 [ 4766 ] {20252ba7-a59e-4b19-a3a7-a3b2afdd4421} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:42:47.244837 [ 3642 ] {} <Error> void DB::SystemLog<DB::MetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::MetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:50.124385 [ 4766 ] {86677bce-2ca5-4931-bf36-ff518ac06fa4} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=11, jobs=11). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:38370) (comment: 00653_verification_monotonic_data_load.sh) (in query: SELECT count() FROM date_test_table WHERE toUInt16(val) == 1;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:42:50.191497 [ 4766 ] {86677bce-2ca5-4931-bf36-ff518ac06fa4} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=11, jobs=11). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:42:50.658281 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_31_6}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:51.590820 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:53.631443 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:54.964318 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/ajn/hclbysqfkullhktshtsogreiewhbw, total size 184549376, count 184549376, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 74991acf-09d5-45c6-9536-818bb9168e4b, upload has finished false.
2024.11.05 19:42:54.972669 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/ajn/hclbysqfkullhktshtsogreiewhbw, total size 184549376, count 184549376, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 74991acf-09d5-45c6-9536-818bb9168e4b, upload has finished false
2024.11.05 19:42:55.057670 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/ajn/hclbysqfkullhktshtsogreiewhbw, total size 184549376, count 184549376, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 74991acf-09d5-45c6-9536-818bb9168e4b, upload has finished false
2024.11.05 19:42:55.350690 [ 4767 ] {ab01ebd8-5acd-4761-a082-f4770272f1ba} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=908, jobs=860). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:38382) (comment: 00145_aggregate_functions_statistics.sql) (in query: SELECT stddevSamp(ResolutionWidth) FROM (SELECT ResolutionWidth FROM test.hits LIMIT 0);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001d5394f3
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:112: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000001d539234
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000001d5395e9
12. /build/src/Server/TCPHandler.cpp:1023: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b57f
13. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:42:55.351874 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 41 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:55.541072 [ 4767 ] {ab01ebd8-5acd-4761-a082-f4770272f1ba} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=908, jobs=860). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001d5394f3
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:112: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000001d539234
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000001d5395e9
12. /build/src/Server/TCPHandler.cpp:1023: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b57f
13. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:42:56.678343 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:42:58.383128 [ 4766 ] {3ae31c4a-1c1e-4cd3-b2b0-3c549124ddd7} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:32950) (comment: 01666_merge_tree_max_query_limit.sh) (in query: select count() > 0 from system.text_log where query_id = 'max_concurrent_queries_25179' and level = 'Warning' and message like '%We have query_id removed but it\'s not recorded. This is a bug%' format TSVRaw), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:42:58.420956 [ 4766 ] {3ae31c4a-1c1e-4cd3-b2b0-3c549124ddd7} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:43:01.974814 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/kfi/owxrpbxxorimghxkuoieyrnervbxe, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 020192ef-5b8b-4fe9-9dba-ae2416c5aa51, upload has finished false.
2024.11.05 19:43:01.977396 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/kfi/owxrpbxxorimghxkuoieyrnervbxe, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 020192ef-5b8b-4fe9-9dba-ae2416c5aa51, upload has finished false
2024.11.05 19:43:02.042543 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/kfi/owxrpbxxorimghxkuoieyrnervbxe, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 020192ef-5b8b-4fe9-9dba-ae2416c5aa51, upload has finished false
2024.11.05 19:43:02.193075 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 15 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:02.322819 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:07.365706 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:07.728120 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {d830f0a7-c4af-4441-961e-a7e3a729e583::202411_1_36_7}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:07.796000 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 9 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:08.322933 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:08.621227 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:12.769191 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:16.097543 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:17.822005 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/ktw/folsgaketdrcnyesuqicqajlvmzrn, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8adf452e-c78b-482c-b54f-635d68480dac, upload has finished false.
2024.11.05 19:43:17.823891 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/ktw/folsgaketdrcnyesuqicqajlvmzrn, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8adf452e-c78b-482c-b54f-635d68480dac, upload has finished false
2024.11.05 19:43:17.952168 [ 3611 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Removing temporary directory /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:43:17.952872 [ 3611 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:43:17.955632 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/ktw/folsgaketdrcnyesuqicqajlvmzrn, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8adf452e-c78b-482c-b54f-635d68480dac, upload has finished false
2024.11.05 19:43:18.026669 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Error> virtual void DB::MetadataStorageFromDiskTransaction::commit(): Code: 521. DB::ErrnoException: Paths cannot be exchanged because /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1 or /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/delete_tmp_tmp_merge_all_1_5_1 does not exist: , errno: 2, strerror: No such file or directory. (ATOMIC_RENAME_FAIL), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:194: DB::ErrnoException::ErrnoException<String const&, String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String const&>::type>, String const&, String const&) @ 0x000000001a0d58cc
6. /build/src/Common/atomicRename.cpp:93: DB::renameat2(String const&, String const&, int) @ 0x000000001a0d53c3
7. /build/src/Common/atomicRename.cpp:211: DB::renameNoReplace(String const&, String const&) @ 0x000000001a0d5102
8. /build/src/Disks/DiskLocal.cpp:320: DB::DiskLocal::moveFile(String const&, String const&) @ 0x000000001a2d53f9
9. /build/src/Disks/ObjectStorages/MetadataStorageFromDiskTransactionOperations.cpp:207: DB::MoveFileOperation::execute(std::unique_lock<DB::SharedMutex>&) @ 0x000000001a3af5cc
10. /build/src/Disks/ObjectStorages/MetadataStorageFromDisk.cpp:186: DB::MetadataStorageFromDiskTransaction::commit() @ 0x000000001a3a369a
11. /build/src/Disks/ObjectStorages/DiskObjectStorageTransaction.cpp:925: DB::DiskObjectStorageTransaction::commit() @ 0x000000001a340c97
12. /build/src/Disks/ObjectStorages/DiskObjectStorage.cpp:197: DB::DiskObjectStorage::moveFile(String const&, String const&, bool) @ 0x000000001a319589
13. /build/src/Disks/ObjectStorages/DiskObjectStorage.cpp:227: DB::DiskObjectStorage::moveFile(String const&, String const&) @ 0x000000001a319a50
14. /build/src/Disks/ObjectStorages/DiskObjectStorage.h:122: DB::DiskObjectStorage::moveDirectory(String const&, String const&) @ 0x000000001a3231a9
15. /build/src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp:666: DB::DataPartStorageOnDiskBase::remove(std::function<DB::CanRemoveDescription ()>&&, DB::MergeTreeDataPartChecksums const&, std::list<DB::IDataPartStorage::ProjectionChecksums, std::allocator<DB::IDataPartStorage::ProjectionChecksums>>, bool, std::shared_ptr<Poco::Logger>) @ 0x000000001ca91998
16. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:1959: DB::IMergeTreeDataPart::remove() @ 0x000000001cab4ce8
17. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:548: DB::IMergeTreeDataPart::removeIfNeeded() @ 0x000000001cab32c7
18. /build/src/Storages/MergeTree/MergeTreeDataPartWide.cpp:189: DB::MergeTreeDataPartWide::~MergeTreeDataPartWide() @ 0x000000001cccc463
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:64: void std::__destroy_at[abi:v15000]<DB::MergeTreeDataPartWide, 0>(DB::MergeTreeDataPartWide*) @ 0x000000001ccb508e
20. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:89: void std::destroy_at[abi:v15000]<DB::MergeTreeDataPartWide, 0>(DB::MergeTreeDataPartWide*) @ 0x000000001ccb5035
21. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:321: void std::allocator_traits<std::allocator<DB::MergeTreeDataPartWide>>::destroy[abi:v15000]<DB::MergeTreeDataPartWide, void, void>(std::allocator<DB::MergeTreeDataPartWide>&, DB::MergeTreeDataPartWide*) @ 0x000000001ccb5019
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:309: std::__shared_ptr_emplace<DB::MergeTreeDataPartWide, std::allocator<DB::MergeTreeDataPartWide>>::__on_zero_shared() @ 0x000000001ccb4e6e
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:175: std::__shared_count::__release_shared[abi:v15000]() @ 0x000000000a8f6c14
24. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215: std::__shared_weak_count::__release_shared[abi:v15000]() @ 0x000000000a8f6bb9
25. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:703: std::shared_ptr<DB::IMergeTreeDataPart>::~shared_ptr[abi:v15000]() @ 0x000000001ba2ab0c
26. /build/src/Storages/MergeTree/IMergeTreeDataPartWriter.cpp:63: DB::IMergeTreeDataPartWriter::~IMergeTreeDataPartWriter() @ 0x000000001ccdf932
27. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.h:43: DB::MergeTreeDataPartWriterOnDisk::~MergeTreeDataPartWriterOnDisk() @ 0x000000001ccd6037
28. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.h:20: DB::MergeTreeDataPartWriterWide::~MergeTreeDataPartWriterWide() @ 0x000000001ccf8d37
29. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.h:20: DB::MergeTreeDataPartWriterWide::~MergeTreeDataPartWriterWide() @ 0x000000001ccf8d59
30. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:49: std::default_delete<DB::IMergeTreeDataPartWriter>::operator()[abi:v15000](DB::IMergeTreeDataPartWriter*) const @ 0x000000001cb7ccf4
31. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:306: std::unique_ptr<DB::IMergeTreeDataPartWriter, std::default_delete<DB::IMergeTreeDataPartWriter>>::reset[abi:v15000](DB::IMergeTreeDataPartWriter*) @ 0x000000001cb7cc5c
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:18.039940 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Error> void DB::IMergeTreeDataPart::removeIfNeeded(): while removing part all_1_5_1 with path store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/: Code: 521. DB::ErrnoException: Paths cannot be exchanged because /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/delete_tmp_tmp_merge_all_1_5_1 or /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1 does not exist: , errno: 2, strerror: No such file or directory: While rolling back operation #0. (ATOMIC_RENAME_FAIL), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:194: DB::ErrnoException::ErrnoException<String const&, String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String const&>::type>, String const&, String const&) @ 0x000000001a0d58cc
6. /build/src/Common/atomicRename.cpp:93: DB::renameat2(String const&, String const&, int) @ 0x000000001a0d53c3
7. /build/src/Common/atomicRename.cpp:211: DB::renameNoReplace(String const&, String const&) @ 0x000000001a0d5102
8. /build/src/Disks/DiskLocal.cpp:320: DB::DiskLocal::moveFile(String const&, String const&) @ 0x000000001a2d53f9
9. /build/src/Disks/ObjectStorages/MetadataStorageFromDiskTransactionOperations.cpp:212: DB::MoveFileOperation::undo() @ 0x000000001a3af608
10. /build/src/Disks/ObjectStorages/MetadataStorageFromDisk.cpp:224: DB::MetadataStorageFromDiskTransaction::rollback(unsigned long) @ 0x000000001a3a3c78
11. /build/src/Disks/ObjectStorages/MetadataStorageFromDisk.cpp:193: DB::MetadataStorageFromDiskTransaction::commit() @ 0x000000001a3a38ab
12. /build/src/Disks/ObjectStorages/DiskObjectStorageTransaction.cpp:925: DB::DiskObjectStorageTransaction::commit() @ 0x000000001a340c97
13. /build/src/Disks/ObjectStorages/DiskObjectStorage.cpp:197: DB::DiskObjectStorage::moveFile(String const&, String const&, bool) @ 0x000000001a319589
14. /build/src/Disks/ObjectStorages/DiskObjectStorage.cpp:227: DB::DiskObjectStorage::moveFile(String const&, String const&) @ 0x000000001a319a50
15. /build/src/Disks/ObjectStorages/DiskObjectStorage.h:122: DB::DiskObjectStorage::moveDirectory(String const&, String const&) @ 0x000000001a3231a9
16. /build/src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp:666: DB::DataPartStorageOnDiskBase::remove(std::function<DB::CanRemoveDescription ()>&&, DB::MergeTreeDataPartChecksums const&, std::list<DB::IDataPartStorage::ProjectionChecksums, std::allocator<DB::IDataPartStorage::ProjectionChecksums>>, bool, std::shared_ptr<Poco::Logger>) @ 0x000000001ca91998
17. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:1959: DB::IMergeTreeDataPart::remove() @ 0x000000001cab4ce8
18. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:548: DB::IMergeTreeDataPart::removeIfNeeded() @ 0x000000001cab32c7
19. /build/src/Storages/MergeTree/MergeTreeDataPartWide.cpp:189: DB::MergeTreeDataPartWide::~MergeTreeDataPartWide() @ 0x000000001cccc463
20. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:64: void std::__destroy_at[abi:v15000]<DB::MergeTreeDataPartWide, 0>(DB::MergeTreeDataPartWide*) @ 0x000000001ccb508e
21. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:89: void std::destroy_at[abi:v15000]<DB::MergeTreeDataPartWide, 0>(DB::MergeTreeDataPartWide*) @ 0x000000001ccb5035
22. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:321: void std::allocator_traits<std::allocator<DB::MergeTreeDataPartWide>>::destroy[abi:v15000]<DB::MergeTreeDataPartWide, void, void>(std::allocator<DB::MergeTreeDataPartWide>&, DB::MergeTreeDataPartWide*) @ 0x000000001ccb5019
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:309: std::__shared_ptr_emplace<DB::MergeTreeDataPartWide, std::allocator<DB::MergeTreeDataPartWide>>::__on_zero_shared() @ 0x000000001ccb4e6e
24. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:175: std::__shared_count::__release_shared[abi:v15000]() @ 0x000000000a8f6c14
25. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215: std::__shared_weak_count::__release_shared[abi:v15000]() @ 0x000000000a8f6bb9
26. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:703: std::shared_ptr<DB::IMergeTreeDataPart>::~shared_ptr[abi:v15000]() @ 0x000000001ba2ab0c
27. /build/src/Storages/MergeTree/IMergeTreeDataPartWriter.cpp:63: DB::IMergeTreeDataPartWriter::~IMergeTreeDataPartWriter() @ 0x000000001ccdf932
28. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.h:43: DB::MergeTreeDataPartWriterOnDisk::~MergeTreeDataPartWriterOnDisk() @ 0x000000001ccd6037
29. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.h:20: DB::MergeTreeDataPartWriterWide::~MergeTreeDataPartWriterWide() @ 0x000000001ccf8d37
30. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.h:20: DB::MergeTreeDataPartWriterWide::~MergeTreeDataPartWriterWide() @ 0x000000001ccf8d59
31. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:49: std::default_delete<DB::IMergeTreeDataPartWriter>::operator()[abi:v15000](DB::IMergeTreeDataPartWriter*) const @ 0x000000001cb7ccf4
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:18.044806 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 28 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:19.405961 [ 4766 ] {285e04fa-e6d2-4f24-aaf1-8b502a396708} <Error> executeQuery: Code: 15. DB::Exception: Column 'tupleElement(CAST(tuple(1), 'Tuple(a Int)'), 1)' already exists: While processing CAST(tuple(1), 'Tuple(a Int)').1, untuple(CAST(tuple(1), 'Tuple(a Int)')). (DUPLICATE_COLUMN) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37802) (comment: 02890_untuple_column_names.sql) (in query: SELECT untuple(tuple(1)::Tuple(a Int)), untuple(tuple(1)::Tuple(a Int)) FORMAT Vertical SETTINGS allow_experimental_analyzer = 0;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Interpreters/ActionsVisitor.cpp:454: DB::ScopeStack::Index::addNode(DB::ActionsDAG::Node const*) @ 0x000000001a7153e1
7. /build/src/Interpreters/ActionsVisitor.cpp:646: DB::ScopeStack::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<String, std::allocator<String>> const&, String) @ 0x000000001a709a68
8. /build/src/Interpreters/ActionsVisitor.h:182: DB::ActionsMatcher::Data::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<String, std::allocator<String>> const&, String) @ 0x000000001a715b21
9. /build/src/Interpreters/ActionsVisitor.cpp:788: DB::ActionsMatcher::doUntuple(DB::ASTFunction const*, DB::ActionsMatcher::Data&) @ 0x000000001a710a1e
10. /build/src/Interpreters/ActionsVisitor.cpp:807: DB::ActionsMatcher::visit(DB::ASTExpressionList&, std::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0x000000001a70fae6
11. /build/src/Interpreters/ActionsVisitor.cpp:708: DB::ActionsMatcher::visit(std::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0x000000001a709f43
12. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::doVisit(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6fff60
13. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visitImplMain<false>(std::shared_ptr<DB::IAST> const&) @ 0x000000001a700141
14. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visitImpl<false>(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6ffeea
15. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visit(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6ddd45
16. /build/src/Interpreters/ExpressionAnalyzer.cpp:484: DB::ExpressionAnalyzer::getRootActions(std::shared_ptr<DB::IAST> const&, bool, std::shared_ptr<DB::ActionsDAG>&, bool) @ 0x000000001a6caabd
17. /build/src/Interpreters/ExpressionAnalyzer.cpp:1511: DB::SelectQueryExpressionAnalyzer::appendSelect(DB::ExpressionActionsChain&, bool) @ 0x000000001a6d4e15
18. /build/src/Interpreters/ExpressionAnalyzer.cpp:2067: DB::ExpressionAnalysisResult::ExpressionAnalysisResult(DB::SelectQueryExpressionAnalyzer&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, bool, bool, bool, std::shared_ptr<DB::FilterDAGInfo> const&, std::shared_ptr<DB::FilterDAGInfo> const&, DB::Block const&) @ 0x000000001a6d9a2a
19. /build/src/Interpreters/InterpreterSelectQuery.cpp:1059: DB::InterpreterSelectQuery::getSampleBlockImpl() @ 0x000000001b455084
20. /build/src/Interpreters/InterpreterSelectQuery.cpp:799: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x000000001b44a93e
21. /build/src/Interpreters/InterpreterSelectQuery.cpp:809: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x000000001b445a42
22. /build/src/Interpreters/InterpreterSelectQuery.cpp:210: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b440c86
23. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQuery, std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::vector<String, std::allocator<String>> const&>(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e8061
24. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:255: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e53c9
25. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:153: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e4845
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b5ebff4
27. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:418: DB::registerInterpreterSelectWithUnionQuery(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b5e74ac
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5e745d
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b5e73fd
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b5e73bd
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b5e7385

2024.11.05 19:43:19.469167 [ 4766 ] {285e04fa-e6d2-4f24-aaf1-8b502a396708} <Error> TCPHandler: Code: 15. DB::Exception: Column 'tupleElement(CAST(tuple(1), 'Tuple(a Int)'), 1)' already exists: While processing CAST(tuple(1), 'Tuple(a Int)').1, untuple(CAST(tuple(1), 'Tuple(a Int)')). (DUPLICATE_COLUMN), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Interpreters/ActionsVisitor.cpp:454: DB::ScopeStack::Index::addNode(DB::ActionsDAG::Node const*) @ 0x000000001a7153e1
7. /build/src/Interpreters/ActionsVisitor.cpp:646: DB::ScopeStack::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<String, std::allocator<String>> const&, String) @ 0x000000001a709a68
8. /build/src/Interpreters/ActionsVisitor.h:182: DB::ActionsMatcher::Data::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<String, std::allocator<String>> const&, String) @ 0x000000001a715b21
9. /build/src/Interpreters/ActionsVisitor.cpp:788: DB::ActionsMatcher::doUntuple(DB::ASTFunction const*, DB::ActionsMatcher::Data&) @ 0x000000001a710a1e
10. /build/src/Interpreters/ActionsVisitor.cpp:807: DB::ActionsMatcher::visit(DB::ASTExpressionList&, std::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0x000000001a70fae6
11. /build/src/Interpreters/ActionsVisitor.cpp:708: DB::ActionsMatcher::visit(std::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0x000000001a709f43
12. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::doVisit(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6fff60
13. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visitImplMain<false>(std::shared_ptr<DB::IAST> const&) @ 0x000000001a700141
14. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visitImpl<false>(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6ffeea
15. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visit(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6ddd45
16. /build/src/Interpreters/ExpressionAnalyzer.cpp:484: DB::ExpressionAnalyzer::getRootActions(std::shared_ptr<DB::IAST> const&, bool, std::shared_ptr<DB::ActionsDAG>&, bool) @ 0x000000001a6caabd
17. /build/src/Interpreters/ExpressionAnalyzer.cpp:1511: DB::SelectQueryExpressionAnalyzer::appendSelect(DB::ExpressionActionsChain&, bool) @ 0x000000001a6d4e15
18. /build/src/Interpreters/ExpressionAnalyzer.cpp:2067: DB::ExpressionAnalysisResult::ExpressionAnalysisResult(DB::SelectQueryExpressionAnalyzer&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, bool, bool, bool, std::shared_ptr<DB::FilterDAGInfo> const&, std::shared_ptr<DB::FilterDAGInfo> const&, DB::Block const&) @ 0x000000001a6d9a2a
19. /build/src/Interpreters/InterpreterSelectQuery.cpp:1059: DB::InterpreterSelectQuery::getSampleBlockImpl() @ 0x000000001b455084
20. /build/src/Interpreters/InterpreterSelectQuery.cpp:799: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x000000001b44a93e
21. /build/src/Interpreters/InterpreterSelectQuery.cpp:809: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x000000001b445a42
22. /build/src/Interpreters/InterpreterSelectQuery.cpp:210: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b440c86
23. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQuery, std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::vector<String, std::allocator<String>> const&>(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e8061
24. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:255: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e53c9
25. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:153: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e4845
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b5ebff4
27. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:418: DB::registerInterpreterSelectWithUnionQuery(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b5e74ac
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5e745d
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b5e73fd
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b5e73bd
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b5e7385

2024.11.05 19:43:20.630279 [ 4766 ] {e0ff7362-cd0c-467f-9e8d-bd96c5c4689e} <Error> executeQuery: Code: 15. DB::Exception: Column 'tupleElement(CAST(tuple(1), 'Tuple(a Int)'), 1)' already exists: While processing CAST(tuple(1), 'Tuple(a Int)').1 AS `x.a`, untuple(CAST(tuple(1), 'Tuple(a Int)')) AS x. (DUPLICATE_COLUMN) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37802) (comment: 02890_untuple_column_names.sql) (in query: SELECT untuple(tuple(1)::Tuple(a Int)) x, untuple(tuple(1)::Tuple(a Int)) x FORMAT Vertical SETTINGS allow_experimental_analyzer = 0;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Interpreters/ActionsVisitor.cpp:454: DB::ScopeStack::Index::addNode(DB::ActionsDAG::Node const*) @ 0x000000001a7153e1
7. /build/src/Interpreters/ActionsVisitor.cpp:646: DB::ScopeStack::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<String, std::allocator<String>> const&, String) @ 0x000000001a709a68
8. /build/src/Interpreters/ActionsVisitor.h:182: DB::ActionsMatcher::Data::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<String, std::allocator<String>> const&, String) @ 0x000000001a715b21
9. /build/src/Interpreters/ActionsVisitor.cpp:788: DB::ActionsMatcher::doUntuple(DB::ASTFunction const*, DB::ActionsMatcher::Data&) @ 0x000000001a710a1e
10. /build/src/Interpreters/ActionsVisitor.cpp:807: DB::ActionsMatcher::visit(DB::ASTExpressionList&, std::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0x000000001a70fae6
11. /build/src/Interpreters/ActionsVisitor.cpp:708: DB::ActionsMatcher::visit(std::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0x000000001a709f43
12. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::doVisit(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6fff60
13. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visitImplMain<false>(std::shared_ptr<DB::IAST> const&) @ 0x000000001a700141
14. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visitImpl<false>(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6ffeea
15. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visit(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6ddd45
16. /build/src/Interpreters/ExpressionAnalyzer.cpp:484: DB::ExpressionAnalyzer::getRootActions(std::shared_ptr<DB::IAST> const&, bool, std::shared_ptr<DB::ActionsDAG>&, bool) @ 0x000000001a6caabd
17. /build/src/Interpreters/ExpressionAnalyzer.cpp:1511: DB::SelectQueryExpressionAnalyzer::appendSelect(DB::ExpressionActionsChain&, bool) @ 0x000000001a6d4e15
18. /build/src/Interpreters/ExpressionAnalyzer.cpp:2067: DB::ExpressionAnalysisResult::ExpressionAnalysisResult(DB::SelectQueryExpressionAnalyzer&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, bool, bool, bool, std::shared_ptr<DB::FilterDAGInfo> const&, std::shared_ptr<DB::FilterDAGInfo> const&, DB::Block const&) @ 0x000000001a6d9a2a
19. /build/src/Interpreters/InterpreterSelectQuery.cpp:1059: DB::InterpreterSelectQuery::getSampleBlockImpl() @ 0x000000001b455084
20. /build/src/Interpreters/InterpreterSelectQuery.cpp:799: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x000000001b44a93e
21. /build/src/Interpreters/InterpreterSelectQuery.cpp:809: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x000000001b445a42
22. /build/src/Interpreters/InterpreterSelectQuery.cpp:210: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b440c86
23. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQuery, std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::vector<String, std::allocator<String>> const&>(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e8061
24. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:255: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e53c9
25. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:153: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e4845
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b5ebff4
27. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:418: DB::registerInterpreterSelectWithUnionQuery(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b5e74ac
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5e745d
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b5e73fd
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b5e73bd
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b5e7385

2024.11.05 19:43:20.652349 [ 4766 ] {e0ff7362-cd0c-467f-9e8d-bd96c5c4689e} <Error> TCPHandler: Code: 15. DB::Exception: Column 'tupleElement(CAST(tuple(1), 'Tuple(a Int)'), 1)' already exists: While processing CAST(tuple(1), 'Tuple(a Int)').1 AS `x.a`, untuple(CAST(tuple(1), 'Tuple(a Int)')) AS x. (DUPLICATE_COLUMN), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Interpreters/ActionsVisitor.cpp:454: DB::ScopeStack::Index::addNode(DB::ActionsDAG::Node const*) @ 0x000000001a7153e1
7. /build/src/Interpreters/ActionsVisitor.cpp:646: DB::ScopeStack::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<String, std::allocator<String>> const&, String) @ 0x000000001a709a68
8. /build/src/Interpreters/ActionsVisitor.h:182: DB::ActionsMatcher::Data::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<String, std::allocator<String>> const&, String) @ 0x000000001a715b21
9. /build/src/Interpreters/ActionsVisitor.cpp:788: DB::ActionsMatcher::doUntuple(DB::ASTFunction const*, DB::ActionsMatcher::Data&) @ 0x000000001a710a1e
10. /build/src/Interpreters/ActionsVisitor.cpp:807: DB::ActionsMatcher::visit(DB::ASTExpressionList&, std::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0x000000001a70fae6
11. /build/src/Interpreters/ActionsVisitor.cpp:708: DB::ActionsMatcher::visit(std::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0x000000001a709f43
12. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::doVisit(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6fff60
13. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visitImplMain<false>(std::shared_ptr<DB::IAST> const&) @ 0x000000001a700141
14. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visitImpl<false>(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6ffeea
15. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visit(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6ddd45
16. /build/src/Interpreters/ExpressionAnalyzer.cpp:484: DB::ExpressionAnalyzer::getRootActions(std::shared_ptr<DB::IAST> const&, bool, std::shared_ptr<DB::ActionsDAG>&, bool) @ 0x000000001a6caabd
17. /build/src/Interpreters/ExpressionAnalyzer.cpp:1511: DB::SelectQueryExpressionAnalyzer::appendSelect(DB::ExpressionActionsChain&, bool) @ 0x000000001a6d4e15
18. /build/src/Interpreters/ExpressionAnalyzer.cpp:2067: DB::ExpressionAnalysisResult::ExpressionAnalysisResult(DB::SelectQueryExpressionAnalyzer&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, bool, bool, bool, std::shared_ptr<DB::FilterDAGInfo> const&, std::shared_ptr<DB::FilterDAGInfo> const&, DB::Block const&) @ 0x000000001a6d9a2a
19. /build/src/Interpreters/InterpreterSelectQuery.cpp:1059: DB::InterpreterSelectQuery::getSampleBlockImpl() @ 0x000000001b455084
20. /build/src/Interpreters/InterpreterSelectQuery.cpp:799: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x000000001b44a93e
21. /build/src/Interpreters/InterpreterSelectQuery.cpp:809: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x000000001b445a42
22. /build/src/Interpreters/InterpreterSelectQuery.cpp:210: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b440c86
23. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQuery, std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::vector<String, std::allocator<String>> const&>(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e8061
24. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:255: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e53c9
25. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:153: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e4845
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b5ebff4
27. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:418: DB::registerInterpreterSelectWithUnionQuery(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b5e74ac
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5e745d
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b5e73fd
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b5e73bd
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b5e7385

2024.11.05 19:43:20.960743 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:21.616903 [ 4766 ] {8d6b7309-dbf5-46be-886f-bb90d3832d38} <Error> executeQuery: Code: 15. DB::Exception: Column 'tupleElement(CAST(tuple(1), 'Tuple(Int)'), 1)' already exists: While processing CAST(tuple(1), 'Tuple(Int)').1 AS `x.1`, untuple(CAST(tuple(1), 'Tuple(Int)')) AS x. (DUPLICATE_COLUMN) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37802) (comment: 02890_untuple_column_names.sql) (in query: SELECT untuple(tuple(1)::Tuple(Int)) x, untuple(tuple(1)::Tuple(Int)) x FORMAT Vertical SETTINGS allow_experimental_analyzer = 0;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Interpreters/ActionsVisitor.cpp:454: DB::ScopeStack::Index::addNode(DB::ActionsDAG::Node const*) @ 0x000000001a7153e1
7. /build/src/Interpreters/ActionsVisitor.cpp:646: DB::ScopeStack::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<String, std::allocator<String>> const&, String) @ 0x000000001a709a68
8. /build/src/Interpreters/ActionsVisitor.h:182: DB::ActionsMatcher::Data::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<String, std::allocator<String>> const&, String) @ 0x000000001a715b21
9. /build/src/Interpreters/ActionsVisitor.cpp:788: DB::ActionsMatcher::doUntuple(DB::ASTFunction const*, DB::ActionsMatcher::Data&) @ 0x000000001a710a1e
10. /build/src/Interpreters/ActionsVisitor.cpp:807: DB::ActionsMatcher::visit(DB::ASTExpressionList&, std::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0x000000001a70fae6
11. /build/src/Interpreters/ActionsVisitor.cpp:708: DB::ActionsMatcher::visit(std::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0x000000001a709f43
12. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::doVisit(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6fff60
13. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visitImplMain<false>(std::shared_ptr<DB::IAST> const&) @ 0x000000001a700141
14. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visitImpl<false>(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6ffeea
15. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visit(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6ddd45
16. /build/src/Interpreters/ExpressionAnalyzer.cpp:484: DB::ExpressionAnalyzer::getRootActions(std::shared_ptr<DB::IAST> const&, bool, std::shared_ptr<DB::ActionsDAG>&, bool) @ 0x000000001a6caabd
17. /build/src/Interpreters/ExpressionAnalyzer.cpp:1511: DB::SelectQueryExpressionAnalyzer::appendSelect(DB::ExpressionActionsChain&, bool) @ 0x000000001a6d4e15
18. /build/src/Interpreters/ExpressionAnalyzer.cpp:2067: DB::ExpressionAnalysisResult::ExpressionAnalysisResult(DB::SelectQueryExpressionAnalyzer&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, bool, bool, bool, std::shared_ptr<DB::FilterDAGInfo> const&, std::shared_ptr<DB::FilterDAGInfo> const&, DB::Block const&) @ 0x000000001a6d9a2a
19. /build/src/Interpreters/InterpreterSelectQuery.cpp:1059: DB::InterpreterSelectQuery::getSampleBlockImpl() @ 0x000000001b455084
20. /build/src/Interpreters/InterpreterSelectQuery.cpp:799: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x000000001b44a93e
21. /build/src/Interpreters/InterpreterSelectQuery.cpp:809: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x000000001b445a42
22. /build/src/Interpreters/InterpreterSelectQuery.cpp:210: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b440c86
23. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQuery, std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::vector<String, std::allocator<String>> const&>(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e8061
24. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:255: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e53c9
25. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:153: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e4845
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b5ebff4
27. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:418: DB::registerInterpreterSelectWithUnionQuery(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b5e74ac
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5e745d
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b5e73fd
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b5e73bd
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b5e7385

2024.11.05 19:43:21.630457 [ 4766 ] {8d6b7309-dbf5-46be-886f-bb90d3832d38} <Error> TCPHandler: Code: 15. DB::Exception: Column 'tupleElement(CAST(tuple(1), 'Tuple(Int)'), 1)' already exists: While processing CAST(tuple(1), 'Tuple(Int)').1 AS `x.1`, untuple(CAST(tuple(1), 'Tuple(Int)')) AS x. (DUPLICATE_COLUMN), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Interpreters/ActionsVisitor.cpp:454: DB::ScopeStack::Index::addNode(DB::ActionsDAG::Node const*) @ 0x000000001a7153e1
7. /build/src/Interpreters/ActionsVisitor.cpp:646: DB::ScopeStack::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<String, std::allocator<String>> const&, String) @ 0x000000001a709a68
8. /build/src/Interpreters/ActionsVisitor.h:182: DB::ActionsMatcher::Data::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<String, std::allocator<String>> const&, String) @ 0x000000001a715b21
9. /build/src/Interpreters/ActionsVisitor.cpp:788: DB::ActionsMatcher::doUntuple(DB::ASTFunction const*, DB::ActionsMatcher::Data&) @ 0x000000001a710a1e
10. /build/src/Interpreters/ActionsVisitor.cpp:807: DB::ActionsMatcher::visit(DB::ASTExpressionList&, std::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0x000000001a70fae6
11. /build/src/Interpreters/ActionsVisitor.cpp:708: DB::ActionsMatcher::visit(std::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0x000000001a709f43
12. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::doVisit(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6fff60
13. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visitImplMain<false>(std::shared_ptr<DB::IAST> const&) @ 0x000000001a700141
14. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visitImpl<false>(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6ffeea
15. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visit(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6ddd45
16. /build/src/Interpreters/ExpressionAnalyzer.cpp:484: DB::ExpressionAnalyzer::getRootActions(std::shared_ptr<DB::IAST> const&, bool, std::shared_ptr<DB::ActionsDAG>&, bool) @ 0x000000001a6caabd
17. /build/src/Interpreters/ExpressionAnalyzer.cpp:1511: DB::SelectQueryExpressionAnalyzer::appendSelect(DB::ExpressionActionsChain&, bool) @ 0x000000001a6d4e15
18. /build/src/Interpreters/ExpressionAnalyzer.cpp:2067: DB::ExpressionAnalysisResult::ExpressionAnalysisResult(DB::SelectQueryExpressionAnalyzer&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, bool, bool, bool, std::shared_ptr<DB::FilterDAGInfo> const&, std::shared_ptr<DB::FilterDAGInfo> const&, DB::Block const&) @ 0x000000001a6d9a2a
19. /build/src/Interpreters/InterpreterSelectQuery.cpp:1059: DB::InterpreterSelectQuery::getSampleBlockImpl() @ 0x000000001b455084
20. /build/src/Interpreters/InterpreterSelectQuery.cpp:799: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x000000001b44a93e
21. /build/src/Interpreters/InterpreterSelectQuery.cpp:809: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x000000001b445a42
22. /build/src/Interpreters/InterpreterSelectQuery.cpp:210: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b440c86
23. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQuery, std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::vector<String, std::allocator<String>> const&>(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e8061
24. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:255: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e53c9
25. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:153: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e4845
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b5ebff4
27. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:418: DB::registerInterpreterSelectWithUnionQuery(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b5e74ac
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5e745d
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b5e73fd
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b5e73bd
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b5e7385

2024.11.05 19:43:22.907865 [ 4766 ] {ca5dd6bd-1fb9-457f-921e-7622077437fb} <Error> executeQuery: Code: 15. DB::Exception: Column 'tupleElement(CAST(tuple(1), 'Tuple(Int)'), 1)' already exists: While processing CAST(tuple(1), 'Tuple(Int)').1, untuple(CAST(tuple(1), 'Tuple(Int)')). (DUPLICATE_COLUMN) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37802) (comment: 02890_untuple_column_names.sql) (in query: SELECT untuple(tuple(1)::Tuple(Int)), untuple(tuple(1)::Tuple(Int)) FORMAT Vertical SETTINGS allow_experimental_analyzer = 0;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Interpreters/ActionsVisitor.cpp:454: DB::ScopeStack::Index::addNode(DB::ActionsDAG::Node const*) @ 0x000000001a7153e1
7. /build/src/Interpreters/ActionsVisitor.cpp:646: DB::ScopeStack::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<String, std::allocator<String>> const&, String) @ 0x000000001a709a68
8. /build/src/Interpreters/ActionsVisitor.h:182: DB::ActionsMatcher::Data::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<String, std::allocator<String>> const&, String) @ 0x000000001a715b21
9. /build/src/Interpreters/ActionsVisitor.cpp:788: DB::ActionsMatcher::doUntuple(DB::ASTFunction const*, DB::ActionsMatcher::Data&) @ 0x000000001a710a1e
10. /build/src/Interpreters/ActionsVisitor.cpp:807: DB::ActionsMatcher::visit(DB::ASTExpressionList&, std::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0x000000001a70fae6
11. /build/src/Interpreters/ActionsVisitor.cpp:708: DB::ActionsMatcher::visit(std::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0x000000001a709f43
12. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::doVisit(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6fff60
13. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visitImplMain<false>(std::shared_ptr<DB::IAST> const&) @ 0x000000001a700141
14. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visitImpl<false>(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6ffeea
15. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visit(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6ddd45
16. /build/src/Interpreters/ExpressionAnalyzer.cpp:484: DB::ExpressionAnalyzer::getRootActions(std::shared_ptr<DB::IAST> const&, bool, std::shared_ptr<DB::ActionsDAG>&, bool) @ 0x000000001a6caabd
17. /build/src/Interpreters/ExpressionAnalyzer.cpp:1511: DB::SelectQueryExpressionAnalyzer::appendSelect(DB::ExpressionActionsChain&, bool) @ 0x000000001a6d4e15
18. /build/src/Interpreters/ExpressionAnalyzer.cpp:2067: DB::ExpressionAnalysisResult::ExpressionAnalysisResult(DB::SelectQueryExpressionAnalyzer&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, bool, bool, bool, std::shared_ptr<DB::FilterDAGInfo> const&, std::shared_ptr<DB::FilterDAGInfo> const&, DB::Block const&) @ 0x000000001a6d9a2a
19. /build/src/Interpreters/InterpreterSelectQuery.cpp:1059: DB::InterpreterSelectQuery::getSampleBlockImpl() @ 0x000000001b455084
20. /build/src/Interpreters/InterpreterSelectQuery.cpp:799: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x000000001b44a93e
21. /build/src/Interpreters/InterpreterSelectQuery.cpp:809: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x000000001b445a42
22. /build/src/Interpreters/InterpreterSelectQuery.cpp:210: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b440c86
23. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQuery, std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::vector<String, std::allocator<String>> const&>(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e8061
24. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:255: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e53c9
25. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:153: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e4845
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b5ebff4
27. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:418: DB::registerInterpreterSelectWithUnionQuery(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b5e74ac
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5e745d
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b5e73fd
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b5e73bd
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b5e7385

2024.11.05 19:43:22.974501 [ 4766 ] {ca5dd6bd-1fb9-457f-921e-7622077437fb} <Error> TCPHandler: Code: 15. DB::Exception: Column 'tupleElement(CAST(tuple(1), 'Tuple(Int)'), 1)' already exists: While processing CAST(tuple(1), 'Tuple(Int)').1, untuple(CAST(tuple(1), 'Tuple(Int)')). (DUPLICATE_COLUMN), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Interpreters/ActionsVisitor.cpp:454: DB::ScopeStack::Index::addNode(DB::ActionsDAG::Node const*) @ 0x000000001a7153e1
7. /build/src/Interpreters/ActionsVisitor.cpp:646: DB::ScopeStack::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<String, std::allocator<String>> const&, String) @ 0x000000001a709a68
8. /build/src/Interpreters/ActionsVisitor.h:182: DB::ActionsMatcher::Data::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<String, std::allocator<String>> const&, String) @ 0x000000001a715b21
9. /build/src/Interpreters/ActionsVisitor.cpp:788: DB::ActionsMatcher::doUntuple(DB::ASTFunction const*, DB::ActionsMatcher::Data&) @ 0x000000001a710a1e
10. /build/src/Interpreters/ActionsVisitor.cpp:807: DB::ActionsMatcher::visit(DB::ASTExpressionList&, std::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0x000000001a70fae6
11. /build/src/Interpreters/ActionsVisitor.cpp:708: DB::ActionsMatcher::visit(std::shared_ptr<DB::IAST> const&, DB::ActionsMatcher::Data&) @ 0x000000001a709f43
12. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::doVisit(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6fff60
13. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visitImplMain<false>(std::shared_ptr<DB::IAST> const&) @ 0x000000001a700141
14. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visitImpl<false>(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6ffeea
15. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor<DB::ActionsMatcher, true, false, std::shared_ptr<DB::IAST> const>::visit(std::shared_ptr<DB::IAST> const&) @ 0x000000001a6ddd45
16. /build/src/Interpreters/ExpressionAnalyzer.cpp:484: DB::ExpressionAnalyzer::getRootActions(std::shared_ptr<DB::IAST> const&, bool, std::shared_ptr<DB::ActionsDAG>&, bool) @ 0x000000001a6caabd
17. /build/src/Interpreters/ExpressionAnalyzer.cpp:1511: DB::SelectQueryExpressionAnalyzer::appendSelect(DB::ExpressionActionsChain&, bool) @ 0x000000001a6d4e15
18. /build/src/Interpreters/ExpressionAnalyzer.cpp:2067: DB::ExpressionAnalysisResult::ExpressionAnalysisResult(DB::SelectQueryExpressionAnalyzer&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, bool, bool, bool, std::shared_ptr<DB::FilterDAGInfo> const&, std::shared_ptr<DB::FilterDAGInfo> const&, DB::Block const&) @ 0x000000001a6d9a2a
19. /build/src/Interpreters/InterpreterSelectQuery.cpp:1059: DB::InterpreterSelectQuery::getSampleBlockImpl() @ 0x000000001b455084
20. /build/src/Interpreters/InterpreterSelectQuery.cpp:799: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x000000001b44a93e
21. /build/src/Interpreters/InterpreterSelectQuery.cpp:809: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x000000001b445a42
22. /build/src/Interpreters/InterpreterSelectQuery.cpp:210: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b440c86
23. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQuery, std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::vector<String, std::allocator<String>> const&>(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e8061
24. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:255: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e53c9
25. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:153: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5e4845
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b5ebff4
27. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:418: DB::registerInterpreterSelectWithUnionQuery(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b5e74ac
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5e745d
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b5e73fd
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b5e73bd
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b5e7385

2024.11.05 19:43:24.942118 [ 3641 ] {} <Error> void DB::SystemLog<DB::PartLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::PartLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:28.655026 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {dbd666d3-2d23-40c6-8136-6ca2be336b4d::202411_1_31_5}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:29.962804 [ 4766 ] {5d0cdf94-9976-457c-8ca8-83945be7d911} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=8, jobs=8). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:55046) (comment: 02475_analyzer_join_tree_subquery.sql) (in query: WITH subquery AS (SELECT sum(number) FROM numbers(10)) SELECT (SELECT * FROM subquery);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:43:29.971090 [ 4766 ] {5d0cdf94-9976-457c-8ca8-83945be7d911} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=8, jobs=8). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:43:31.426237 [ 4767 ] {d9c319d9-d3b6-4f01-9721-90d0a8863a4b} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:55054) (comment: 00940_order_by_read_in_order.sql) (in query: INSERT INTO pk_order(a, b, c, d) VALUES ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6

2024.11.05 19:43:31.437115 [ 4767 ] {d9c319d9-d3b6-4f01-9721-90d0a8863a4b} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6

2024.11.05 19:43:32.340995 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/lnc/pwexggyrlumjlyluyotrgatrnriqj, total size 201326592, count 201326592, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 12d5b005-fa51-4db8-ac26-65e09fe1fc36, upload has finished false.
2024.11.05 19:43:32.343901 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/lnc/pwexggyrlumjlyluyotrgatrnriqj, total size 201326592, count 201326592, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 12d5b005-fa51-4db8-ac26-65e09fe1fc36, upload has finished false
2024.11.05 19:43:32.481006 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/lnc/pwexggyrlumjlyluyotrgatrnriqj, total size 201326592, count 201326592, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 12d5b005-fa51-4db8-ac26-65e09fe1fc36, upload has finished false
2024.11.05 19:43:32.552291 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 44 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:39.897643 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:43.028740 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:47.875022 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/ukh/ysptprortupsmkfduzmpbocracfde, total size 218103808, count 218103808, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id e8747ade-c192-4bdf-b561-e634c22f3a34, upload has finished false.
2024.11.05 19:43:47.880199 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/ukh/ysptprortupsmkfduzmpbocracfde, total size 218103808, count 218103808, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id e8747ade-c192-4bdf-b561-e634c22f3a34, upload has finished false
2024.11.05 19:43:47.890197 [ 2816 ] {3a690e0b-e90a-48df-9186-e9db146390f9} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 19.599513802 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:55034) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:43:47.967641 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/ukh/ysptprortupsmkfduzmpbocracfde, total size 218103808, count 218103808, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id e8747ade-c192-4bdf-b561-e634c22f3a34, upload has finished false
2024.11.05 19:43:47.991619 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 46 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:49.287829 [ 2816 ] {3a690e0b-e90a-48df-9186-e9db146390f9} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 19.599513802 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:43:49.842465 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:52.477490 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0a233324-63e4-4351-8154-d23ef4927e18::202411_1_31_6}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
24. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
25. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
26. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
27. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:53.669688 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {29fdad69-1244-4c25-af3c-70c43e910f15::202411_1_61_12}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:43:59.528676 [ 4766 ] {e04f6b87-cb93-4acb-acbc-680786a4adfb} <Error> executeQuery: Code: 164. DB::Exception: default: Cannot execute query in readonly mode. (READONLY) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46850) (comment: 02473_functions_in_readonly_mode.sh) (in query: SELECT * from format('TSV', '123')), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Access/ContextAccess.cpp:546: auto bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessFlags) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x00000000195756ca
7. /build/src/Access/ContextAccess.cpp:627: bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessFlags) const @ 0x000000001957315b
8. /build/src/Access/ContextAccess.cpp:651: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessFlags const&) const @ 0x000000001953f982
9. /build/src/Access/ContextAccess.cpp:724: DB::ContextAccess::checkAccess(DB::AccessFlags const&) const @ 0x000000001953adfd
10. /build/src/Interpreters/Context.cpp:1398: void DB::Context::checkAccessImpl<DB::AccessFlags>(DB::AccessFlags const&) const @ 0x000000001a56f756
11. /build/src/Interpreters/Context.cpp:1401: DB::Context::checkAccess(DB::AccessFlags const&) const @ 0x000000001a544fbd
12. /build/src/TableFunctions/ITableFunction.cpp:32: DB::ITableFunction::execute(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, String const&, DB::ColumnsDescription, bool, bool) const @ 0x00000000190c5d7a
13. /build/src/Interpreters/Context.cpp:2014: DB::Context::executeTableFunction(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::ITableFunction> const&) @ 0x000000001a54a848
14. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:7558: DB::(anonymous namespace)::QueryAnalyzer::resolveTableFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, DB::(anonymous namespace)::QueryExpressionsAliasVisitor&, bool) @ 0x000000001b0a6a29
15. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:7917: DB::(anonymous namespace)::QueryAnalyzer::resolveQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, DB::(anonymous namespace)::QueryExpressionsAliasVisitor&) @ 0x000000001b0ab4b8
16. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8164: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4d6
17. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
18. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
19. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
20. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
21. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
22. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
23. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
30. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
31. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0

2024.11.05 19:43:59.573638 [ 4766 ] {e04f6b87-cb93-4acb-acbc-680786a4adfb} <Error> TCPHandler: Code: 164. DB::Exception: default: Cannot execute query in readonly mode. (READONLY), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Access/ContextAccess.cpp:546: auto bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessFlags) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x00000000195756ca
7. /build/src/Access/ContextAccess.cpp:627: bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessFlags) const @ 0x000000001957315b
8. /build/src/Access/ContextAccess.cpp:651: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessFlags const&) const @ 0x000000001953f982
9. /build/src/Access/ContextAccess.cpp:724: DB::ContextAccess::checkAccess(DB::AccessFlags const&) const @ 0x000000001953adfd
10. /build/src/Interpreters/Context.cpp:1398: void DB::Context::checkAccessImpl<DB::AccessFlags>(DB::AccessFlags const&) const @ 0x000000001a56f756
11. /build/src/Interpreters/Context.cpp:1401: DB::Context::checkAccess(DB::AccessFlags const&) const @ 0x000000001a544fbd
12. /build/src/TableFunctions/ITableFunction.cpp:32: DB::ITableFunction::execute(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, String const&, DB::ColumnsDescription, bool, bool) const @ 0x00000000190c5d7a
13. /build/src/Interpreters/Context.cpp:2014: DB::Context::executeTableFunction(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::ITableFunction> const&) @ 0x000000001a54a848
14. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:7558: DB::(anonymous namespace)::QueryAnalyzer::resolveTableFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, DB::(anonymous namespace)::QueryExpressionsAliasVisitor&, bool) @ 0x000000001b0a6a29
15. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:7917: DB::(anonymous namespace)::QueryAnalyzer::resolveQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, DB::(anonymous namespace)::QueryExpressionsAliasVisitor&) @ 0x000000001b0ab4b8
16. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8164: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4d6
17. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
18. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
19. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
20. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
21. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
22. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
23. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
30. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
31. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0

2024.11.05 19:43:59.733286 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/dyn/inrzodlcogbgoyjjrrfbbwztdisaj, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 909c1523-d875-49e1-9728-bb033016b25d, upload has finished false.
2024.11.05 19:43:59.735941 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/dyn/inrzodlcogbgoyjjrrfbbwztdisaj, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 909c1523-d875-49e1-9728-bb033016b25d, upload has finished false
2024.11.05 19:43:59.793919 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/dyn/inrzodlcogbgoyjjrrfbbwztdisaj, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 909c1523-d875-49e1-9728-bb033016b25d, upload has finished false
2024.11.05 19:43:59.949377 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 28 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:03.482875 [ 2816 ] {5a0fc854-b321-4c74-9bf9-83bb8ea62267} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 9.811240599 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46826) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:44:03.988432 [ 4766 ] {be277837-82d1-4acd-b759-559fdf1bae58} <Error> executeQuery: Code: 164. DB::Exception: default: Cannot execute query in readonly mode. (READONLY) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:57474) (comment: 02473_functions_in_readonly_mode.sh) (in query: INSERT INTO FUNCTION null('x String') (x) FORMAT TSV ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Access/ContextAccess.cpp:546: auto bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessFlags) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x00000000195756ca
7. /build/src/Access/ContextAccess.cpp:627: bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessFlags) const @ 0x000000001957315b
8. /build/src/Access/ContextAccess.cpp:651: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessFlags const&) const @ 0x000000001953f982
9. /build/src/Access/ContextAccess.cpp:724: DB::ContextAccess::checkAccess(DB::AccessFlags const&) const @ 0x000000001953adfd
10. /build/src/Interpreters/Context.cpp:1398: void DB::Context::checkAccessImpl<DB::AccessFlags>(DB::AccessFlags const&) const @ 0x000000001a56f756
11. /build/src/Interpreters/Context.cpp:1401: DB::Context::checkAccess(DB::AccessFlags const&) const @ 0x000000001a544fbd
12. /build/src/TableFunctions/ITableFunction.cpp:32: DB::ITableFunction::execute(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, String const&, DB::ColumnsDescription, bool, bool) const @ 0x00000000190c5d7a
13. /build/src/Interpreters/InterpreterInsertQuery.cpp:107: DB::InterpreterInsertQuery::getTable(DB::ASTInsertQuery&) @ 0x000000001b429b9e
14. /build/src/Interpreters/InterpreterInsertQuery.cpp:406: DB::InterpreterInsertQuery::execute() @ 0x000000001b42c384
15. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
16. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
17. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
18. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
19. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
20. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
21. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
22. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
23. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
24. ? @ 0x00007f3c91188ac3
25. ? @ 0x00007f3c9121a850

2024.11.05 19:44:04.012245 [ 4766 ] {be277837-82d1-4acd-b759-559fdf1bae58} <Error> TCPHandler: Code: 164. DB::Exception: default: Cannot execute query in readonly mode. (READONLY), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Access/ContextAccess.cpp:546: auto bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessFlags) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x00000000195756ca
7. /build/src/Access/ContextAccess.cpp:627: bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessFlags) const @ 0x000000001957315b
8. /build/src/Access/ContextAccess.cpp:651: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessFlags const&) const @ 0x000000001953f982
9. /build/src/Access/ContextAccess.cpp:724: DB::ContextAccess::checkAccess(DB::AccessFlags const&) const @ 0x000000001953adfd
10. /build/src/Interpreters/Context.cpp:1398: void DB::Context::checkAccessImpl<DB::AccessFlags>(DB::AccessFlags const&) const @ 0x000000001a56f756
11. /build/src/Interpreters/Context.cpp:1401: DB::Context::checkAccess(DB::AccessFlags const&) const @ 0x000000001a544fbd
12. /build/src/TableFunctions/ITableFunction.cpp:32: DB::ITableFunction::execute(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, String const&, DB::ColumnsDescription, bool, bool) const @ 0x00000000190c5d7a
13. /build/src/Interpreters/InterpreterInsertQuery.cpp:107: DB::InterpreterInsertQuery::getTable(DB::ASTInsertQuery&) @ 0x000000001b429b9e
14. /build/src/Interpreters/InterpreterInsertQuery.cpp:406: DB::InterpreterInsertQuery::execute() @ 0x000000001b42c384
15. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
16. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
17. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
18. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
19. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
20. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
21. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
22. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
23. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
24. ? @ 0x00007f3c91188ac3
25. ? @ 0x00007f3c9121a850

2024.11.05 19:44:04.568144 [ 2816 ] {5a0fc854-b321-4c74-9bf9-83bb8ea62267} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 9.811240599 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:44:12.060922 [ 5196 ] {a8b535ee-894d-4862-9b9f-015f53d17249} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=917, jobs=860). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60026) (comment: 02242_case_insensitive_column_matching.sh) (in query: SELECT * FROM orc_load), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001d5394f3
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:112: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000001d539234
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000001d5395e9
12. /build/src/Server/TCPHandler.cpp:1023: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b57f
13. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:44:12.082531 [ 5196 ] {a8b535ee-894d-4862-9b9f-015f53d17249} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=917, jobs=860). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001d5394f3
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:112: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000001d539234
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000001d5395e9
12. /build/src/Server/TCPHandler.cpp:1023: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b57f
13. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:44:16.649034 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/hax/ctwbvvdxvitvuqcqcigfgxvaccset, total size 251658240, count 251658240, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 82614082-0cb3-4819-a653-95d76b147ad8, upload has finished false.
2024.11.05 19:44:16.657340 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/hax/ctwbvvdxvitvuqcqcigfgxvaccset, total size 251658240, count 251658240, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 82614082-0cb3-4819-a653-95d76b147ad8, upload has finished false
2024.11.05 19:44:16.748175 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/hax/ctwbvvdxvitvuqcqcigfgxvaccset, total size 251658240, count 251658240, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 82614082-0cb3-4819-a653-95d76b147ad8, upload has finished false
2024.11.05 19:44:16.776259 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 54 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:17.009775 [ 4766 ] {673e138c-83b1-43f8-b618-8e36bc6e33dc} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 8.189953109 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:57508) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:44:17.483025 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:17.527959 [ 3580 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key s3/mnb/ysmubmrftjnjmhbtpofssoyharhsw, total size 122843301, count 122843301, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id c35f729c-65d4-4ef5-90bb-a30680312b7a, upload has finished false.
2024.11.05 19:44:17.528854 [ 3580 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key s3/mnb/ysmubmrftjnjmhbtpofssoyharhsw, total size 122843301, count 122843301, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id c35f729c-65d4-4ef5-90bb-a30680312b7a, upload has finished false
2024.11.05 19:44:17.600452 [ 3580 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key s3/mnb/ysmubmrftjnjmhbtpofssoyharhsw, total size 122843301, count 122843301, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id c35f729c-65d4-4ef5-90bb-a30680312b7a, upload has finished false
2024.11.05 19:44:17.662874 [ 3580 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key s3/egh/cvrjcxxcaounpaykwmleggvhjbxzn, total size 44068035, count 44068035, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id 5679af43-ee78-45fd-9830-654426c09db5, upload has finished false.
2024.11.05 19:44:17.665063 [ 3580 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key s3/egh/cvrjcxxcaounpaykwmleggvhjbxzn, total size 44068035, count 44068035, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id 5679af43-ee78-45fd-9830-654426c09db5, upload has finished false
2024.11.05 19:44:17.699402 [ 3580 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key s3/egh/cvrjcxxcaounpaykwmleggvhjbxzn, total size 44068035, count 44068035, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id 5679af43-ee78-45fd-9830-654426c09db5, upload has finished false
2024.11.05 19:44:17.882781 [ 4766 ] {673e138c-83b1-43f8-b618-8e36bc6e33dc} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 8.189953109 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:44:18.123114 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:21.286486 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 6 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:22.273438 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:24.476441 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=6). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:26.117266 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:29.855642 [ 4766 ] {380a36e5-f9a2-4baf-ab80-dee7bd18e531} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 7.628834577 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:45852) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:44:30.382554 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {d830f0a7-c4af-4441-961e-a7e3a729e583::202411_1_46_9}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:30.895317 [ 4766 ] {380a36e5-f9a2-4baf-ab80-dee7bd18e531} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 7.628834577 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:44:30.912408 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:34.828362 [ 2816 ] {47cd7688-00be-430a-891d-2a6e8c8ae4d8} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=35, jobs=35). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:57494) (comment: 01493_alter_remove_properties_zookeeper.sql) (in query: SELECT COUNT() FROM r_prop_table1;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:44:34.857428 [ 2816 ] {47cd7688-00be-430a-891d-2a6e8c8ae4d8} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=35, jobs=35). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:44:37.571249 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:39.101902 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/uxa/lfltiifbevldmfvqogkzvvwaoktnk, total size 251658240, count 251658240, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f311d3ab-f3e6-46d0-b1ea-dfa0b48a7347, upload has finished false.
2024.11.05 19:44:39.103021 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/uxa/lfltiifbevldmfvqogkzvvwaoktnk, total size 251658240, count 251658240, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f311d3ab-f3e6-46d0-b1ea-dfa0b48a7347, upload has finished false
2024.11.05 19:44:39.190742 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/uxa/lfltiifbevldmfvqogkzvvwaoktnk, total size 251658240, count 251658240, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f311d3ab-f3e6-46d0-b1ea-dfa0b48a7347, upload has finished false
2024.11.05 19:44:39.278672 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 54 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:43.738955 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:44.868261 [ 2816 ] {835273f9-715e-4b1b-9961-7b52d056c931} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 9.409749288 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48842) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:44:45.680691 [ 4766 ] {c83100cd-db0b-4107-9ab1-f555de2c23cd} <Error> executeQuery: Code: 107. DB::Exception: File /var/lib/clickhouse/user_files/nonexist.txt doesn't exist. (FILE_DOESNT_EXIST) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48538) (in query: select _path,_file from file('nonexist.txt', 'CSV', 'val1 char')), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Storages/StorageFile.cpp:1573: DB::StorageFile::read(DB::QueryPlan&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::SelectQueryInfo&, std::shared_ptr<DB::Context const>, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000001c37b16d
7. /build/src/Planner/PlannerJoinTree.cpp:878: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr<DB::IQueryTreeNode>, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::PlannerContext>&, bool, bool) @ 0x000000001b553818
8. /build/src/Planner/PlannerJoinTree.cpp:1667: DB::buildJoinTreeQueryPlan(std::shared_ptr<DB::IQueryTreeNode> const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>> const&, std::shared_ptr<DB::PlannerContext>&) @ 0x000000001b54ef10
9. /build/src/Planner/Planner.cpp:1473: DB::Planner::buildPlanForQueryNode() @ 0x000000001b50fd06
10. /build/src/Planner/Planner.cpp:0: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000001b50be55
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:234: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000001b50905d
12. /build/src/Interpreters/executeQuery.cpp:1146: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2812
13. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
14. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
15. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:44:45.691354 [ 4766 ] {c83100cd-db0b-4107-9ab1-f555de2c23cd} <Error> TCPHandler: Code: 107. DB::Exception: File /var/lib/clickhouse/user_files/nonexist.txt doesn't exist. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Storages/StorageFile.cpp:1573: DB::StorageFile::read(DB::QueryPlan&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::SelectQueryInfo&, std::shared_ptr<DB::Context const>, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000001c37b16d
7. /build/src/Planner/PlannerJoinTree.cpp:878: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr<DB::IQueryTreeNode>, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::PlannerContext>&, bool, bool) @ 0x000000001b553818
8. /build/src/Planner/PlannerJoinTree.cpp:1667: DB::buildJoinTreeQueryPlan(std::shared_ptr<DB::IQueryTreeNode> const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>> const&, std::shared_ptr<DB::PlannerContext>&) @ 0x000000001b54ef10
9. /build/src/Planner/Planner.cpp:1473: DB::Planner::buildPlanForQueryNode() @ 0x000000001b50fd06
10. /build/src/Planner/Planner.cpp:0: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000001b50be55
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:234: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000001b50905d
12. /build/src/Interpreters/executeQuery.cpp:1146: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2812
13. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
14. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
15. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:44:45.770682 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:45.848962 [ 2816 ] {835273f9-715e-4b1b-9961-7b52d056c931} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 9.409749288 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:44:49.722668 [ 2816 ] {1febb7c6-bbea-40c6-85d0-168664667f26} <Error> executeQuery: Code: 636. DB::Exception: The table structure cannot be extracted from a Parquet format file. Error:
Code: 50. DB::Exception: Unsupported Parquet type 'null' of an input column 'error_message'. If it happens during schema inference and you want to skip columns with unsupported types, you can enable setting input_format_parquet_skip_columns_with_unsupported_types_in_schema_inference. (UNKNOWN_TYPE) (version 24.3.12.76.altinitystable (altinity build)).
You can specify the structure manually: (in file/uri /var/lib/clickhouse/user_files/test_02245.parquet). (CANNOT_EXTRACT_TABLE_STRUCTURE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48558) (comment: 02245_parquet_skip_unknown_type.sh) (in query: desc file(test_02245.parquet)), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&) @ 0x0000000011aa53ec
6. /build/src/Formats/ReadSchemaUtils.cpp:279: DB::readSchemaFromFormatImpl(std::optional<String>, std::optional<DB::FormatSettings> const&, DB::IReadBufferIterator&, std::shared_ptr<DB::Context const> const&) @ 0x000000001d4d0e9b
7. /build/src/Formats/ReadSchemaUtils.cpp:541: DB::readSchemaFromFormat(String const&, std::optional<DB::FormatSettings> const&, DB::IReadBufferIterator&, std::shared_ptr<DB::Context const> const&) @ 0x000000001d4d3d06
8. /build/src/Storages/StorageFile.cpp:943: DB::StorageFile::getTableStructureAndFormatFromFileImpl(std::optional<String>, std::vector<String, std::allocator<String>> const&, String const&, std::optional<DB::FormatSettings> const&, std::shared_ptr<DB::Context const> const&, std::optional<DB::StorageFile::ArchiveInfo> const&) @ 0x000000001c372cbe
9. /build/src/Storages/StorageFile.cpp:956: DB::StorageFile::getTableStructureFromFile(String const&, std::vector<String, std::allocator<String>> const&, String const&, std::optional<DB::FormatSettings> const&, std::shared_ptr<DB::Context const> const&, std::optional<DB::StorageFile::ArchiveInfo> const&) @ 0x000000001c37313c
10. /build/src/TableFunctions/TableFunctionFile.cpp:109: DB::TableFunctionFile::getActualTableStructure(std::shared_ptr<DB::Context const>, bool) const @ 0x0000000018d4b92b
11. /build/src/Interpreters/InterpreterDescribeQuery.cpp:150: DB::InterpreterDescribeQuery::fillColumnsFromTableFunction(DB::ASTTableExpression const&) @ 0x000000001bc239a1
12. /build/src/Interpreters/InterpreterDescribeQuery.cpp:91: DB::InterpreterDescribeQuery::execute() @ 0x000000001bc22f1f
13. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
14. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
15. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
16. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850

2024.11.05 19:44:49.739940 [ 2816 ] {1febb7c6-bbea-40c6-85d0-168664667f26} <Error> TCPHandler: Code: 636. DB::Exception: The table structure cannot be extracted from a Parquet format file. Error:
Code: 50. DB::Exception: Unsupported Parquet type 'null' of an input column 'error_message'. If it happens during schema inference and you want to skip columns with unsupported types, you can enable setting input_format_parquet_skip_columns_with_unsupported_types_in_schema_inference. (UNKNOWN_TYPE) (version 24.3.12.76.altinitystable (altinity build)).
You can specify the structure manually: (in file/uri /var/lib/clickhouse/user_files/test_02245.parquet). (CANNOT_EXTRACT_TABLE_STRUCTURE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&) @ 0x0000000011aa53ec
6. /build/src/Formats/ReadSchemaUtils.cpp:279: DB::readSchemaFromFormatImpl(std::optional<String>, std::optional<DB::FormatSettings> const&, DB::IReadBufferIterator&, std::shared_ptr<DB::Context const> const&) @ 0x000000001d4d0e9b
7. /build/src/Formats/ReadSchemaUtils.cpp:541: DB::readSchemaFromFormat(String const&, std::optional<DB::FormatSettings> const&, DB::IReadBufferIterator&, std::shared_ptr<DB::Context const> const&) @ 0x000000001d4d3d06
8. /build/src/Storages/StorageFile.cpp:943: DB::StorageFile::getTableStructureAndFormatFromFileImpl(std::optional<String>, std::vector<String, std::allocator<String>> const&, String const&, std::optional<DB::FormatSettings> const&, std::shared_ptr<DB::Context const> const&, std::optional<DB::StorageFile::ArchiveInfo> const&) @ 0x000000001c372cbe
9. /build/src/Storages/StorageFile.cpp:956: DB::StorageFile::getTableStructureFromFile(String const&, std::vector<String, std::allocator<String>> const&, String const&, std::optional<DB::FormatSettings> const&, std::shared_ptr<DB::Context const> const&, std::optional<DB::StorageFile::ArchiveInfo> const&) @ 0x000000001c37313c
10. /build/src/TableFunctions/TableFunctionFile.cpp:109: DB::TableFunctionFile::getActualTableStructure(std::shared_ptr<DB::Context const>, bool) const @ 0x0000000018d4b92b
11. /build/src/Interpreters/InterpreterDescribeQuery.cpp:150: DB::InterpreterDescribeQuery::fillColumnsFromTableFunction(DB::ASTTableExpression const&) @ 0x000000001bc239a1
12. /build/src/Interpreters/InterpreterDescribeQuery.cpp:91: DB::InterpreterDescribeQuery::execute() @ 0x000000001bc22f1f
13. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
14. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
15. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
16. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850

2024.11.05 19:44:52.593083 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/gqg/fnnupsxzdjxdrwnmoxjkciknvjrqa, total size 184549376, count 184549376, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0e575b8d-20f1-4b43-ab1b-884adda52fa2, upload has finished false.
2024.11.05 19:44:52.585690 [ 4766 ] {87436bb1-d492-485a-8294-344a01cd8f8c} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.316342588 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48568) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:44:52.610393 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/gqg/fnnupsxzdjxdrwnmoxjkciknvjrqa, total size 184549376, count 184549376, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0e575b8d-20f1-4b43-ab1b-884adda52fa2, upload has finished false
2024.11.05 19:44:52.695991 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/gqg/fnnupsxzdjxdrwnmoxjkciknvjrqa, total size 184549376, count 184549376, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0e575b8d-20f1-4b43-ab1b-884adda52fa2, upload has finished false
2024.11.05 19:44:52.808854 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 39 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:52.874278 [ 4766 ] {87436bb1-d492-485a-8294-344a01cd8f8c} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.316342588 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:44:55.933136 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=1): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 4 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:57.969363 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:44:58.802197 [ 4766 ] {86342bc7-fbd8-4282-92f6-c5e49e160d5b} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.582217983 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37654) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:44:58.997201 [ 4766 ] {86342bc7-fbd8-4282-92f6-c5e49e160d5b} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.582217983 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:44:59.031621 [ 5215 ] {5f287c30-4468-4f82-b6b4-bf6a4edb50c7} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=17, jobs=3). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37656) (comment: 02245_parquet_skip_unknown_type.sh) (in query: select count(*) from file(test_02245.parquet) settings input_format_parquet_skip_columns_with_unsupported_types_in_schema_inference=1), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:44:59.183491 [ 5215 ] {5f287c30-4468-4f82-b6b4-bf6a4edb50c7} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=17, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:44:59.864443 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:01.062436 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 9 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:02.491259 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:09.198746 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/fjd/bzidnriqwecyumghxcrnezrpbklfl, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 394e798d-cf51-438b-a846-319d8d2b5881, upload has finished false.
2024.11.05 19:45:09.201773 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/fjd/bzidnriqwecyumghxcrnezrpbklfl, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 394e798d-cf51-438b-a846-319d8d2b5881, upload has finished false
2024.11.05 19:45:09.252000 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/fjd/bzidnriqwecyumghxcrnezrpbklfl, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 394e798d-cf51-438b-a846-319d8d2b5881, upload has finished false
2024.11.05 19:45:09.337080 [ 3613 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Temporary directory store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/ suddenly disappeared while iterating, assuming it was concurrently renamed to persistent
2024.11.05 19:45:09.344564 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 20 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:10.120094 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_46_9}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:11.688774 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:12.503872 [ 4766 ] {} <Error> ServerErrorHandler: Code: 81. DB::Exception: Database no_such_database_could_exist does not exist. (UNKNOWN_DATABASE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Interpreters/DatabaseCatalog.cpp:517: DB::DatabaseCatalog::assertDatabaseExists(String const&) const @ 0x000000001a671901
7. /build/src/Interpreters/Context.cpp:2301: DB::Context::setCurrentDatabaseWithLock(String const&, std::lock_guard<DB::ContextSharedMutex> const&) @ 0x000000001a54468d
8. /build/src/Interpreters/Context.cpp:2309: DB::Context::setCurrentDatabase(String const&) @ 0x000000001a54d5dd
9. /build/src/Server/TCPHandler.cpp:289: DB::TCPHandler::runImpl() @ 0x000000001d46d72b
10. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
11. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
12. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
13. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
14. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
15. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
16. ? @ 0x00007f3c91188ac3
17. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:12.576303 [ 5215 ] {} <Error> ServerErrorHandler: Code: 81. DB::Exception: Database no_such_database_could_exist does not exist. (UNKNOWN_DATABASE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Interpreters/DatabaseCatalog.cpp:517: DB::DatabaseCatalog::assertDatabaseExists(String const&) const @ 0x000000001a671901
7. /build/src/Interpreters/Context.cpp:2301: DB::Context::setCurrentDatabaseWithLock(String const&, std::lock_guard<DB::ContextSharedMutex> const&) @ 0x000000001a54468d
8. /build/src/Interpreters/Context.cpp:2309: DB::Context::setCurrentDatabase(String const&) @ 0x000000001a54d5dd
9. /build/src/Server/TCPHandler.cpp:289: DB::TCPHandler::runImpl() @ 0x000000001d46d72b
10. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
11. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
12. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
13. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
14. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
15. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
16. ? @ 0x00007f3c91188ac3
17. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:13.056979 [ 2816 ] {efe11209-dd86-40c9-a960-80f10a123731} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 9.637717872 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:45748) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:45:14.069850 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0e641ae6-a90e-4cec-8ebd-a21f62e7934a::202411_1_41_8}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:14.280491 [ 2816 ] {efe11209-dd86-40c9-a960-80f10a123731} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 9.637717872 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:45:17.741894 [ 2816 ] {bee0e0da-fbaa-4cf8-8464-3e8f1af3feb6} <Error> executeQuery: Code: 191. DB::Exception: Limit for IN-set exceeded, max rows: 20.00, current rows: 21.00: While executing CreatingSetsTransform. (SET_SIZE_LIMIT_EXCEEDED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60800) (comment: 01134_set_overflow_mode.sql) (in query: SELECT arrayJoin([5, 25]) IN (SELECT DISTINCT toUInt8(intDiv(number, 10)) FROM numbers(300));), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000019ec2928
6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000019ec244c
7. /build/src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x0000000019ec2799
8. /build/src/Interpreters/Set.cpp:253: DB::Set::insertFromColumns(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>> const&, DB::SetKeyColumns&) @ 0x000000001b8d3bd1
9. /build/src/Interpreters/Set.cpp:200: DB::Set::insertFromColumns(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>> const&) @ 0x000000001b8d2cb6
10. /build/src/Interpreters/Set.cpp:188: DB::Set::insertFromBlock(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) @ 0x000000001b8d2bd4
11. /build/src/Processors/Transforms/CreatingSetsTransform.cpp:211: DB::CreatingSetsTransform::consume(DB::Chunk) @ 0x000000001da24788
12. /build/src/Processors/IAccumulatingTransform.cpp:88: DB::IAccumulatingTransform::work() @ 0x000000001d4f1775
13. /build/src/Processors/Transforms/CreatingSetsTransform.cpp:83: DB::CreatingSetsTransform::work() @ 0x000000001da1f4dc
14. /build/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001d5343e3
15. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001d534120
16. /build/src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d5166a1
17. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
18. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
25. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
26. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
28. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
29. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
30. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355

2024.11.05 19:45:17.750435 [ 2816 ] {bee0e0da-fbaa-4cf8-8464-3e8f1af3feb6} <Error> TCPHandler: Code: 191. DB::Exception: Limit for IN-set exceeded, max rows: 20.00, current rows: 21.00: While executing CreatingSetsTransform. (SET_SIZE_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000019ec2928
6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000019ec244c
7. /build/src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x0000000019ec2799
8. /build/src/Interpreters/Set.cpp:253: DB::Set::insertFromColumns(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>> const&, DB::SetKeyColumns&) @ 0x000000001b8d3bd1
9. /build/src/Interpreters/Set.cpp:200: DB::Set::insertFromColumns(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>> const&) @ 0x000000001b8d2cb6
10. /build/src/Interpreters/Set.cpp:188: DB::Set::insertFromBlock(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) @ 0x000000001b8d2bd4
11. /build/src/Processors/Transforms/CreatingSetsTransform.cpp:211: DB::CreatingSetsTransform::consume(DB::Chunk) @ 0x000000001da24788
12. /build/src/Processors/IAccumulatingTransform.cpp:88: DB::IAccumulatingTransform::work() @ 0x000000001d4f1775
13. /build/src/Processors/Transforms/CreatingSetsTransform.cpp:83: DB::CreatingSetsTransform::work() @ 0x000000001da1f4dc
14. /build/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001d5343e3
15. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001d534120
16. /build/src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d5166a1
17. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
18. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
25. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
26. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
28. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
29. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
30. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355

2024.11.05 19:45:18.844265 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/zww/dyesfaavpemfosjmbwtbvptjaguym, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b8ca286a-5670-4abb-8d82-35505c471a33, upload has finished false.
2024.11.05 19:45:18.848221 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/zww/dyesfaavpemfosjmbwtbvptjaguym, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b8ca286a-5670-4abb-8d82-35505c471a33, upload has finished false
2024.11.05 19:45:18.891871 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/zww/dyesfaavpemfosjmbwtbvptjaguym, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b8ca286a-5670-4abb-8d82-35505c471a33, upload has finished false
2024.11.05 19:45:18.941995 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 21 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:19.665184 [ 2816 ] {d9a86648-bdd1-407c-ab4f-a98f1c2b6d4d} <Error> executeQuery: Code: 191. DB::Exception: Limit for IN-set exceeded, max rows: 20.00, current rows: 21.00: While executing CreatingSetsTransform. (SET_SIZE_LIMIT_EXCEEDED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60800) (comment: 01134_set_overflow_mode.sql) (in query: SELECT arrayJoin([5, 25]) IN (SELECT DISTINCT toUInt8(intDiv(number, 10)) FROM numbers(210));), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000019ec2928
6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000019ec244c
7. /build/src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x0000000019ec2799
8. /build/src/Interpreters/Set.cpp:253: DB::Set::insertFromColumns(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>> const&, DB::SetKeyColumns&) @ 0x000000001b8d3bd1
9. /build/src/Interpreters/Set.cpp:200: DB::Set::insertFromColumns(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>> const&) @ 0x000000001b8d2cb6
10. /build/src/Interpreters/Set.cpp:188: DB::Set::insertFromBlock(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) @ 0x000000001b8d2bd4
11. /build/src/Processors/Transforms/CreatingSetsTransform.cpp:211: DB::CreatingSetsTransform::consume(DB::Chunk) @ 0x000000001da24788
12. /build/src/Processors/IAccumulatingTransform.cpp:88: DB::IAccumulatingTransform::work() @ 0x000000001d4f1775
13. /build/src/Processors/Transforms/CreatingSetsTransform.cpp:83: DB::CreatingSetsTransform::work() @ 0x000000001da1f4dc
14. /build/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001d5343e3
15. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001d534120
16. /build/src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d5166a1
17. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
18. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
25. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
26. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
28. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
29. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
30. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355

2024.11.05 19:45:19.671470 [ 2816 ] {d9a86648-bdd1-407c-ab4f-a98f1c2b6d4d} <Error> TCPHandler: Code: 191. DB::Exception: Limit for IN-set exceeded, max rows: 20.00, current rows: 21.00: While executing CreatingSetsTransform. (SET_SIZE_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000019ec2928
6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000019ec244c
7. /build/src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x0000000019ec2799
8. /build/src/Interpreters/Set.cpp:253: DB::Set::insertFromColumns(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>> const&, DB::SetKeyColumns&) @ 0x000000001b8d3bd1
9. /build/src/Interpreters/Set.cpp:200: DB::Set::insertFromColumns(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>> const&) @ 0x000000001b8d2cb6
10. /build/src/Interpreters/Set.cpp:188: DB::Set::insertFromBlock(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) @ 0x000000001b8d2bd4
11. /build/src/Processors/Transforms/CreatingSetsTransform.cpp:211: DB::CreatingSetsTransform::consume(DB::Chunk) @ 0x000000001da24788
12. /build/src/Processors/IAccumulatingTransform.cpp:88: DB::IAccumulatingTransform::work() @ 0x000000001d4f1775
13. /build/src/Processors/Transforms/CreatingSetsTransform.cpp:83: DB::CreatingSetsTransform::work() @ 0x000000001da1f4dc
14. /build/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001d5343e3
15. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001d534120
16. /build/src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d5166a1
17. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
18. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
25. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
26. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
28. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
29. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
30. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355

2024.11.05 19:45:24.533782 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:25.294366 [ 2816 ] {4b688f1f-be6b-4d43-89cc-852defb38c7f} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:34550) (comment: 02332_dist_insert_send_logs_level.sh) (in query: INSERT INTO FUNCTION remote('127.2', currentDatabase(), data_02332) SELECT * FROM numbers(10)), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/Distributed/DistributedSink.cpp:509: DB::DistributedSink::writeSync(DB::Block const&) @ 0x000000001ca46390
10. /build/src/Storages/Distributed/DistributedSink.cpp:148: DB::DistributedSink::consume(DB::Chunk) @ 0x000000001ca458e6
11. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
12. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:114: DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x000000001da41c5a
20. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work() @ 0x000000001da4169e
21. /build/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001d5343e3
22. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001d534120
23. /build/src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d5166a1
24. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
25. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5

2024.11.05 19:45:25.357568 [ 2816 ] {4b688f1f-be6b-4d43-89cc-852defb38c7f} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/Distributed/DistributedSink.cpp:509: DB::DistributedSink::writeSync(DB::Block const&) @ 0x000000001ca46390
10. /build/src/Storages/Distributed/DistributedSink.cpp:148: DB::DistributedSink::consume(DB::Chunk) @ 0x000000001ca458e6
11. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
12. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:114: DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x000000001da41c5a
20. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work() @ 0x000000001da4169e
21. /build/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001d5343e3
22. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001d534120
23. /build/src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d5166a1
24. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
25. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5

2024.11.05 19:45:27.762419 [ 4766 ] {b2ea79df-4257-4b3d-a74c-ef6a1e4b4f61} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 9.197536909 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60812) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9,10}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:45:28.617734 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/xpz/nqekmzohktuxitzuhcslzuvybzymn, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6d4937c2-1470-47f1-9bd0-1e39beb0803f, upload has finished false.
2024.11.05 19:45:28.618356 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/xpz/nqekmzohktuxitzuhcslzuvybzymn, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6d4937c2-1470-47f1-9bd0-1e39beb0803f, upload has finished false
2024.11.05 19:45:28.663353 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/xpz/nqekmzohktuxitzuhcslzuvybzymn, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6d4937c2-1470-47f1-9bd0-1e39beb0803f, upload has finished false
2024.11.05 19:45:28.700539 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 26 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:29.013320 [ 4766 ] {b2ea79df-4257-4b3d-a74c-ef6a1e4b4f61} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 9.197536909 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e96fcc
6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:111: bool DB::handleOverflowMode<double, double>(DB::OverflowMode, int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000019e963d9
7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:127: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000019e962e8
8. /build/src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c5e
9. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
10. /build/src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514a16
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
16. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
24. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:45:30.252653 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:31.120795 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 3 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:33.256157 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:36.408447 [ 3810 ] {dd60bc10-94cd-4f09-b352-01b59503c914} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=917, jobs=872). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:41000) (comment: 01868_order_by_fill_with_datetime64.sql) (in query: SELECT n, source FROM (SELECT toDateTime64(number * 1000, 9,'Asia/Istanbul') AS n, 'original' AS source FROM numbers(10) WHERE (number % 3) = 1 ) ORDER BY n ASC WITH FILL STEP toDateTime64(1000, 9);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001d5394f3
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:112: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000001d539234
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000001d5395e9
12. /build/src/Server/TCPHandler.cpp:1023: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b57f
13. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:36.440043 [ 3810 ] {dd60bc10-94cd-4f09-b352-01b59503c914} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=917, jobs=872). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001d5394f3
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:112: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000001d539234
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000001d5395e9
12. /build/src/Server/TCPHandler.cpp:1023: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b57f
13. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:37.762654 [ 4766 ] {03e052eb-419b-4427-aeba-a22d53560c2d} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:40994) (comment: 00056_view.sql) (in query: SELECT c, count() FROM test.view GROUP BY c ORDER BY count() DESC LIMIT 10;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:45:37.780443 [ 4766 ] {03e052eb-419b-4427-aeba-a22d53560c2d} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:45:39.110701 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/jbq/xbbolmeiotexhhniasvkkuiniyfao, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 95df5e0e-79ce-45cc-9e92-c90d70a8a01a, upload has finished false.
2024.11.05 19:45:39.112668 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/jbq/xbbolmeiotexhhniasvkkuiniyfao, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 95df5e0e-79ce-45cc-9e92-c90d70a8a01a, upload has finished false
2024.11.05 19:45:39.161826 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/jbq/xbbolmeiotexhhniasvkkuiniyfao, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 95df5e0e-79ce-45cc-9e92-c90d70a8a01a, upload has finished false
2024.11.05 19:45:39.230714 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 15 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:40.821286 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:41.353512 [ 2816 ] {228df85a-51e6-4b71-ac65-2e1438915338} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:34564) (comment: 01440_to_date_monotonicity.sql) (in query: INSERT INTO tdm2 VALUES ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6

2024.11.05 19:45:41.364299 [ 2816 ] {228df85a-51e6-4b71-ac65-2e1438915338} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6

2024.11.05 19:45:45.120874 [ 5215 ] {c9d78fad-914f-48d8-ac67-d99cb3fec23f} <Error> executeQuery: Code: 36. DB::Exception: It does not make sense to have codec NONE along with other compression codecs: NONE, NONE. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37044) (comment: 01272_suspicious_codecs.sql) (in query: -- test what should not work
 CREATE TABLE codecs (a UInt8 CODEC(NONE, NONE)) ENGINE = MergeTree ORDER BY tuple();), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:198: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b0394e3
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:45.126717 [ 5215 ] {c9d78fad-914f-48d8-ac67-d99cb3fec23f} <Error> TCPHandler: Code: 36. DB::Exception: It does not make sense to have codec NONE along with other compression codecs: NONE, NONE. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:198: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b0394e3
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:45.236457 [ 5215 ] {585e90aa-54d0-441f-9832-87100f5cb02d} <Error> executeQuery: Code: 36. DB::Exception: It does not make sense to have codec NONE along with other compression codecs: NONE, LZ4. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37044) (comment: 01272_suspicious_codecs.sql) (in query: CREATE TABLE codecs (a UInt8 CODEC(NONE, LZ4)) ENGINE = MergeTree ORDER BY tuple();), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:198: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b0394e3
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:45.251558 [ 5215 ] {585e90aa-54d0-441f-9832-87100f5cb02d} <Error> TCPHandler: Code: 36. DB::Exception: It does not make sense to have codec NONE along with other compression codecs: NONE, LZ4. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:198: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b0394e3
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:45.335830 [ 5215 ] {1b525329-336f-4895-8c8a-425d4c433ec6} <Error> executeQuery: Code: 36. DB::Exception: It does not make sense to have codec NONE along with other compression codecs: LZ4, NONE. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37044) (comment: 01272_suspicious_codecs.sql) (in query: CREATE TABLE codecs (a UInt8 CODEC(LZ4, NONE)) ENGINE = MergeTree ORDER BY tuple();), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:198: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b0394e3
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:45.339714 [ 5215 ] {1b525329-336f-4895-8c8a-425d4c433ec6} <Error> TCPHandler: Code: 36. DB::Exception: It does not make sense to have codec NONE along with other compression codecs: LZ4, NONE. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:198: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b0394e3
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:45.735042 [ 3642 ] {} <Error> void DB::SystemLog<DB::MetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::MetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:45.744209 [ 5215 ] {2402d988-1079-48c4-bc29-867f3362dfc8} <Error> executeQuery: Code: 36. DB::Exception: The combination of compression codecs LZ4, LZ4 is meaningless, because it does not make sense to apply any transformations after generic compression algorithm. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37044) (comment: 01272_suspicious_codecs.sql) (in query: CREATE TABLE codecs (a UInt8 CODEC(LZ4, LZ4)) ENGINE = MergeTree ORDER BY tuple();), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:243: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b039ac9
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:45.752891 [ 5215 ] {2402d988-1079-48c4-bc29-867f3362dfc8} <Error> TCPHandler: Code: 36. DB::Exception: The combination of compression codecs LZ4, LZ4 is meaningless, because it does not make sense to apply any transformations after generic compression algorithm. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:243: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b039ac9
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:45.880118 [ 5215 ] {0c3122e5-fdce-44ca-b9e3-7e2750d4f3b5} <Error> executeQuery: Code: 36. DB::Exception: The combination of compression codecs LZ4, ZSTD(1) is meaningless, because it does not make sense to apply any transformations after generic compression algorithm. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37044) (comment: 01272_suspicious_codecs.sql) (in query: CREATE TABLE codecs (a UInt8 CODEC(LZ4, ZSTD)) ENGINE = MergeTree ORDER BY tuple();), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:243: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b039ac9
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:45.892427 [ 5215 ] {0c3122e5-fdce-44ca-b9e3-7e2750d4f3b5} <Error> TCPHandler: Code: 36. DB::Exception: The combination of compression codecs LZ4, ZSTD(1) is meaningless, because it does not make sense to apply any transformations after generic compression algorithm. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:243: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b039ac9
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:46.235166 [ 5215 ] {67537157-9746-4fb8-aee4-3f170cb3086b} <Error> executeQuery: Code: 36. DB::Exception: Compression codec Delta(1) does not compress anything. You may want to add generic compression algorithm after other transformations, like: Delta(1), LZ4. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37044) (comment: 01272_suspicious_codecs.sql) (in query: CREATE TABLE codecs (a UInt8 CODEC(Delta)) ENGINE = MergeTree ORDER BY tuple();), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&) @ 0x0000000011aa53ec
6. /build/src/Compression/CompressionFactoryAdditions.cpp:207: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b039606
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:46.255142 [ 5215 ] {67537157-9746-4fb8-aee4-3f170cb3086b} <Error> TCPHandler: Code: 36. DB::Exception: Compression codec Delta(1) does not compress anything. You may want to add generic compression algorithm after other transformations, like: Delta(1), LZ4. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&) @ 0x0000000011aa53ec
6. /build/src/Compression/CompressionFactoryAdditions.cpp:207: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b039606
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:46.385813 [ 5215 ] {bb5ab62b-85fa-4690-9fde-2ebfa6f50079} <Error> executeQuery: Code: 36. DB::Exception: Compression codec Delta(1), Delta(1) does not compress anything. You may want to add generic compression algorithm after other transformations, like: Delta(1), Delta(1), LZ4. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37044) (comment: 01272_suspicious_codecs.sql) (in query: CREATE TABLE codecs (a UInt8 CODEC(Delta, Delta)) ENGINE = MergeTree ORDER BY tuple();), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&) @ 0x0000000011aa53ec
6. /build/src/Compression/CompressionFactoryAdditions.cpp:207: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b039606
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:46.393564 [ 5215 ] {bb5ab62b-85fa-4690-9fde-2ebfa6f50079} <Error> TCPHandler: Code: 36. DB::Exception: Compression codec Delta(1), Delta(1) does not compress anything. You may want to add generic compression algorithm after other transformations, like: Delta(1), Delta(1), LZ4. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&) @ 0x0000000011aa53ec
6. /build/src/Compression/CompressionFactoryAdditions.cpp:207: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b039606
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:46.539028 [ 5215 ] {bf88d59d-8075-4ba8-a407-6c4efb6e7781} <Error> executeQuery: Code: 36. DB::Exception: The combination of compression codecs LZ4, Delta(1) is meaningless, because it does not make sense to apply any transformations after generic compression algorithm. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37044) (comment: 01272_suspicious_codecs.sql) (in query: CREATE TABLE codecs (a UInt8 CODEC(LZ4, Delta)) ENGINE = MergeTree ORDER BY tuple();), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:243: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b039ac9
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:46.544795 [ 5215 ] {bf88d59d-8075-4ba8-a407-6c4efb6e7781} <Error> TCPHandler: Code: 36. DB::Exception: The combination of compression codecs LZ4, Delta(1) is meaningless, because it does not make sense to apply any transformations after generic compression algorithm. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:243: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b039ac9
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:46.842990 [ 5215 ] {d14af4cc-f089-425a-9753-137481182037} <Error> executeQuery: Code: 36. DB::Exception: The combination of compression codecs Gorilla is meaningless, because it does not make sense to apply a floating-point time series codec to non-floating-point columns (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37044) (comment: 01272_suspicious_codecs.sql) (in query: CREATE TABLE codecs (a UInt8 CODEC(Gorilla)) ENGINE = MergeTree ORDER BY tuple();), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:225: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b039887
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:46.852171 [ 5215 ] {d14af4cc-f089-425a-9753-137481182037} <Error> TCPHandler: Code: 36. DB::Exception: The combination of compression codecs Gorilla is meaningless, because it does not make sense to apply a floating-point time series codec to non-floating-point columns (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:225: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b039887
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:47.024376 [ 5215 ] {1e77bec4-d036-4a94-af7f-a57a75dc8c43} <Error> executeQuery: Code: 36. DB::Exception: The combination of compression codecs Gorilla is meaningless, because it does not make sense to apply a floating-point time series codec to non-floating-point columns (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37044) (comment: 01272_suspicious_codecs.sql) (in query: CREATE TABLE codecs (a FixedString(2) CODEC(Gorilla)) ENGINE = MergeTree ORDER BY tuple();), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:225: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b039887
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:47.048289 [ 5215 ] {1e77bec4-d036-4a94-af7f-a57a75dc8c43} <Error> TCPHandler: Code: 36. DB::Exception: The combination of compression codecs Gorilla is meaningless, because it does not make sense to apply a floating-point time series codec to non-floating-point columns (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:225: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b039887
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:47.158381 [ 5215 ] {ffb36ecb-8f71-4436-9995-1c7dd4e57e53} <Error> executeQuery: Code: 36. DB::Exception: The combination of compression codecs Gorilla is meaningless, because it does not make sense to apply a floating-point time series codec to non-floating-point columns (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37044) (comment: 01272_suspicious_codecs.sql) (in query: CREATE TABLE codecs (a Decimal(15,5) CODEC(Gorilla)) ENGINE = MergeTree ORDER BY tuple();), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:225: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b039887
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:47.188904 [ 5215 ] {ffb36ecb-8f71-4436-9995-1c7dd4e57e53} <Error> TCPHandler: Code: 36. DB::Exception: The combination of compression codecs Gorilla is meaningless, because it does not make sense to apply a floating-point time series codec to non-floating-point columns (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:225: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b039887
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:47.520641 [ 5215 ] {62ced521-2b4b-4080-8d25-0f8d5b93f660} <Error> executeQuery: Code: 36. DB::Exception: The combination of compression codecs Delta(8), Gorilla is meaningless, because floating point time series codecs do delta compression implicitly by themselves. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37044) (comment: 01272_suspicious_codecs.sql) (in query: CREATE TABLE codecs (a Float64 CODEC(Delta, Gorilla)) ENGINE = MergeTree ORDER BY tuple();), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:234: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b03999f
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:47.527725 [ 5215 ] {62ced521-2b4b-4080-8d25-0f8d5b93f660} <Error> TCPHandler: Code: 36. DB::Exception: The combination of compression codecs Delta(8), Gorilla is meaningless, because floating point time series codecs do delta compression implicitly by themselves. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:234: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b03999f
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:47.692557 [ 5215 ] {ffa945a7-c59c-46b1-bbfa-d1681c23f565} <Error> executeQuery: Code: 36. DB::Exception: The combination of compression codecs DoubleDelta, FPC(12) is meaningless, because floating point time series codecs do delta compression implicitly by themselves. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37044) (comment: 01272_suspicious_codecs.sql) (in query: CREATE TABLE codecs (a Float32 CODEC(DoubleDelta, FPC)) ENGINE = MergeTree ORDER BY tuple();), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:234: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b03999f
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:47.698600 [ 5215 ] {ffa945a7-c59c-46b1-bbfa-d1681c23f565} <Error> TCPHandler: Code: 36. DB::Exception: The combination of compression codecs DoubleDelta, FPC(12) is meaningless, because floating point time series codecs do delta compression implicitly by themselves. (Note: you can enable setting 'allow_suspicious_codecs' to skip this check). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Compression/CompressionFactoryAdditions.cpp:234: DB::CompressionCodecFactory::validateCodecAndGetPreprocessedAST(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::IDataType const> const&, bool, bool, bool, bool) const @ 0x000000001b03999f
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:672: DB::InterpreterCreateQuery::getColumnsDescription(DB::ASTExpressionList const&, std::shared_ptr<DB::Context const>, bool, bool) @ 0x000000001b00f0ec
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:743: DB::InterpreterCreateQuery::getTablePropertiesAndNormalizeCreateQuery(DB::ASTCreateQuery&) const @ 0x000000001b00fb89
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1221: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b0175c4
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:45:48.117286 [ 2816 ] {5c4131ea-e371-40e5-9a91-479689cacf39} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37040) (comment: 00030_array_enumerate_uniq.sql) (in query: SELECT max(arrayJoin(arrayEnumerateUniq(arrayMap(x -> intDiv(x, 10), URLCategories)))) FROM test.hits ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:45:48.280392 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/zyj/ebtekhevafpppuxymgyqyyrkqxgwv, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 55e1318e-d45a-496b-a5a6-054cdb158b00, upload has finished false.
2024.11.05 19:45:48.281592 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/zyj/ebtekhevafpppuxymgyqyyrkqxgwv, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 55e1318e-d45a-496b-a5a6-054cdb158b00, upload has finished false
2024.11.05 19:45:48.328154 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/zyj/ebtekhevafpppuxymgyqyyrkqxgwv, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 55e1318e-d45a-496b-a5a6-054cdb158b00, upload has finished false
2024.11.05 19:45:48.350530 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 27 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:48.528168 [ 2816 ] {5c4131ea-e371-40e5-9a91-479689cacf39} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:45:49.356445 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:53.864870 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:45:58.161711 [ 4766 ] {ba33f11d-774d-455e-8946-542a0d145d4c} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=4). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37200) (comment: 02943_exprs_order_in_group_by_with_rollup.sql) (in query: insert into test_group_by_with_rollup_order values), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6

2024.11.05 19:45:58.172822 [ 4766 ] {ba33f11d-774d-455e-8946-542a0d145d4c} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6

2024.11.05 19:45:59.652778 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:00.112498 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:00.290907 [ 4766 ] {4e2eb2ef-bf62-49af-9650-68cf95aa3961} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37216) (comment: 00159_parallel_formatting_json_and_friends_2.sh) (in query: SELECT ClientEventTime::DateTime('Asia/Dubai') as a, MobilePhoneModel as b, ClientIP6 as c FROM test.hits ORDER BY a, b, c LIMIT 3000000 Format JSONCompactStringsEachRowWithNames), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:00.410225 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {29fdad69-1244-4c25-af3c-70c43e910f15::202411_1_83_16}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
24. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
25. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
26. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
27. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:00.693924 [ 4766 ] {4e2eb2ef-bf62-49af-9650-68cf95aa3961} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:02.061628 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/ouc/jugzsjsvizeaooxuxdwjapwultlkq, total size 184549376, count 184549376, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 95b77583-d04b-46cc-9fbd-cbcbd07eecbf, upload has finished false.
2024.11.05 19:46:02.063023 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/ouc/jugzsjsvizeaooxuxdwjapwultlkq, total size 184549376, count 184549376, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 95b77583-d04b-46cc-9fbd-cbcbd07eecbf, upload has finished false
2024.11.05 19:46:02.136396 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/ouc/jugzsjsvizeaooxuxdwjapwultlkq, total size 184549376, count 184549376, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 95b77583-d04b-46cc-9fbd-cbcbd07eecbf, upload has finished false
2024.11.05 19:46:02.223395 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=1): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 41 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:03.001163 [ 4766 ] {0cc61a46-d278-493c-8a2b-49c5ffd939d3} <Warning> DeflateQplJobHWPool: Initialization of hardware-assisted DeflateQpl codec failed, falling back to software DeflateQpl codec. Failed to get available workqueue size -> total_wq_size: 0, QPL Version: 1.2.0.
2024.11.05 19:46:05.586190 [ 3810 ] {65469744-7ab3-4a78-b9ef-b54cabe29025} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:35634) (comment: 00159_parallel_formatting_json_and_friends_2.sh) (in query: SELECT ClientEventTime::DateTime('Asia/Dubai') as a, MobilePhoneModel as b, ClientIP6 as c FROM test.hits ORDER BY a, b, c LIMIT 3000000 Format JSONCompactStringsEachRowWithNames), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:05.822475 [ 3810 ] {65469744-7ab3-4a78-b9ef-b54cabe29025} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:11.099275 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:11.155432 [ 3810 ] {5be859b8-8c8e-4dda-90c7-4b39abb619f0} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:35642) (comment: 00159_parallel_formatting_json_and_friends_2.sh) (in query: SELECT ClientEventTime::DateTime('Asia/Dubai') as a, MobilePhoneModel as b, ClientIP6 as c FROM test.hits ORDER BY a, b, c LIMIT 3000000 Format JSONCompactEachRowWithNamesAndTypes), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:11.324960 [ 3810 ] {5be859b8-8c8e-4dda-90c7-4b39abb619f0} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:12.642741 [ 4766 ] {de47f6c1-734d-4ed8-9c22-edce3453e112} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:37224) (comment: 00804_test_deflate_qpl_codec_compression.sql) (in query: SELECT count(*) FROM compression_codec WHERE id = 3 GROUP BY id;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:12.692471 [ 4766 ] {de47f6c1-734d-4ed8-9c22-edce3453e112} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:13.785802 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/wev/zmqpzojfofqcmojwktedcaywajfkf, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c54dee7d-1144-4055-a299-3c9d414f0851, upload has finished false.
2024.11.05 19:46:13.790546 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/wev/zmqpzojfofqcmojwktedcaywajfkf, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c54dee7d-1144-4055-a299-3c9d414f0851, upload has finished false
2024.11.05 19:46:13.923684 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/wev/zmqpzojfofqcmojwktedcaywajfkf, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c54dee7d-1144-4055-a299-3c9d414f0851, upload has finished false
2024.11.05 19:46:13.959448 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:567: DB::WriteBufferFromS3::writePart(DB::WriteBufferFromS3::PartData&&) @ 0x0000000019914d6e
18. /build/src/IO/WriteBufferFromS3.cpp:370: DB::WriteBufferFromS3::writeMultipartUpload() @ 0x0000000019908eb6
19. /build/src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::nextImpl() @ 0x000000001990857e
20. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
21. /build/src/IO/WriteBufferFromFileDecorator.cpp:71: DB::WriteBufferFromFileDecorator::nextImpl() @ 0x000000001a3593ad
22. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
23. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
24. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
25. /build/src/IO/WriteBuffer.h:76: DB::WriteBuffer::nextIfAtEnd() @ 0x000000000a931978
26. /build/src/IO/WriteBuffer.h:91: DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000a9317e0
27. /build/src/Compression/CompressedWriteBuffer.cpp:58: DB::CompressedWriteBuffer::nextImpl() @ 0x0000000019ddd7ed
28. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
29. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
30. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
31. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:350: DB::MergeTreeDataPartWriterWide::getCurrentMarksForColumn(DB::NameAndTypePair const&, std::set<String, std::less<String>, std::allocator<String>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ccf5839
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:16.170693 [ 3581 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key s3/rzr/lxmmsxizjgdltszajviqywtsyhyfd, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 66e51724-7aa9-42f7-92e7-363ee1305758, upload has finished false.
2024.11.05 19:46:16.171616 [ 3581 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key s3/rzr/lxmmsxizjgdltszajviqywtsyhyfd, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 66e51724-7aa9-42f7-92e7-363ee1305758, upload has finished false
2024.11.05 19:46:16.217256 [ 3581 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key s3/rzr/lxmmsxizjgdltszajviqywtsyhyfd, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 66e51724-7aa9-42f7-92e7-363ee1305758, upload has finished false
2024.11.05 19:46:16.616903 [ 3642 ] {} <Error> void DB::SystemLog<DB::MetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::MetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:16.629342 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column Referer): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_5_5_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 16 with max_rows_to_read = 6739): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:16.686108 [ 4766 ] {37b3aaec-1577-4df8-bed1-10684eed9550} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:57298) (comment: 00159_parallel_formatting_json_and_friends_2.sh) (in query: SELECT ClientEventTime::DateTime('Asia/Dubai') as a, MobilePhoneModel as b, ClientIP6 as c FROM test.hits ORDER BY a, b, c LIMIT 3000000 Format JSONCompactEachRowWithNamesAndTypes), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:17.104602 [ 4766 ] {37b3aaec-1577-4df8-bed1-10684eed9550} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:17.784889 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {86a06611-889e-455b-b452-b6a6203be276::202411_1_40_8}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:20.567488 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column StartDate): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_1_1_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 0 with max_rows_to_read = 6658): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:22.494915 [ 4766 ] {157d1c22-90cf-42cb-b8b1-b658b423d62a} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:57328) (comment: 00159_parallel_formatting_json_and_friends_2.sh) (in query: SELECT ClientEventTime::DateTime('Asia/Dubai') as a, MobilePhoneModel as b, ClientIP6 as c FROM test.hits ORDER BY a, b, c LIMIT 3000000 Format JSONCompactStringsEachRowWithNamesAndTypes), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:46:23.361689 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/wsp/jnvlbvbovtcrtgcilzqyebztmsgrj, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 96c44566-ad04-462d-8637-999e71339b47, upload has finished false.
2024.11.05 19:46:23.362321 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/wsp/jnvlbvbovtcrtgcilzqyebztmsgrj, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 96c44566-ad04-462d-8637-999e71339b47, upload has finished false
2024.11.05 19:46:23.413502 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/wsp/jnvlbvbovtcrtgcilzqyebztmsgrj, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 96c44566-ad04-462d-8637-999e71339b47, upload has finished false
2024.11.05 19:46:23.505763 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=1): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 19 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:23.648068 [ 4766 ] {157d1c22-90cf-42cb-b8b1-b658b423d62a} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:46:28.349669 [ 5215 ] {d83360a0-2993-4456-84fe-58f9139348e3} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:42782) (comment: 00159_parallel_formatting_json_and_friends_2.sh) (in query: SELECT ClientEventTime::DateTime('Asia/Dubai') as a, MobilePhoneModel as b, ClientIP6 as c FROM test.hits ORDER BY a, b, c LIMIT 3000000 Format JSONCompactStringsEachRowWithNamesAndTypes), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:46:29.217329 [ 5215 ] {d83360a0-2993-4456-84fe-58f9139348e3} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:46:30.344218 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/wlz/xajhzmyhytgzrrgpfhdpslyegjrrg, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8d00f340-4ebf-481c-878f-c60c54e1e60a, upload has finished false.
2024.11.05 19:46:30.344965 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/wlz/xajhzmyhytgzrrgpfhdpslyegjrrg, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8d00f340-4ebf-481c-878f-c60c54e1e60a, upload has finished false
2024.11.05 19:46:30.356616 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/wlz/xajhzmyhytgzrrgpfhdpslyegjrrg, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8d00f340-4ebf-481c-878f-c60c54e1e60a, upload has finished false
2024.11.05 19:46:30.469870 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 12 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:36.285053 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {62ec4990-945b-4d8e-9889-ca33c67862f8::202411_1_56_11}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
25. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
26. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
27. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
28. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:36.547459 [ 3580 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/zgp/aoqmlfizqyxxkwnhsvbxzmgddwqmb, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id cfd726b8-f37e-4bf6-957b-73fa10c653b2, upload has finished false.
2024.11.05 19:46:36.549541 [ 3580 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/zgp/aoqmlfizqyxxkwnhsvbxzmgddwqmb, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id cfd726b8-f37e-4bf6-957b-73fa10c653b2, upload has finished false
2024.11.05 19:46:36.602226 [ 3580 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/zgp/aoqmlfizqyxxkwnhsvbxzmgddwqmb, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id cfd726b8-f37e-4bf6-957b-73fa10c653b2, upload has finished false
2024.11.05 19:46:36.689550 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 13 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:39.714110 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:40.428263 [ 2816 ] {d65cad68-a151-406b-aace-af4f835b17ef} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=49, jobs=49). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47758) (comment: 00051_min_max_array.sql) (in query: SELECT CounterID, count(), max(GoalsReached), min(GoalsReached), minIf(GoalsReached, notEmpty(GoalsReached)) FROM test.hits GROUP BY CounterID ORDER BY count() DESC LIMIT 20;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:43.246954 [ 2816 ] {d65cad68-a151-406b-aace-af4f835b17ef} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=49, jobs=49). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:45.226178 [ 5793 ] {f814997b-08e3-4156-90dd-ef96ee0f0ff3} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:36130) (comment: 02046_low_cardinality_parallel_group_by.sh) (in query: SELECT count() FROM low_card_buffer_test GROUP BY test_text format Null), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:45.297161 [ 5793 ] {f814997b-08e3-4156-90dd-ef96ee0f0ff3} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:50.158005 [ 3810 ] {153a92ac-efb7-47bb-b597-46476f9e1e75} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:36172) (comment: 00016_any_if_distributed_cond_always_false.sql) (in query: SELECT anyIf(SearchPhrase, CounterID = -1) FROM remote('127.0.0.{1,2}:9000', test, hits) ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:50.257515 [ 3810 ] {153a92ac-efb7-47bb-b597-46476f9e1e75} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:51.076259 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/saf/xkaoqimfhptjbzdctejwxxhtoqeit, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f885a432-ef61-4df6-a73b-f318a2e16d86, upload has finished false.
2024.11.05 19:46:51.077972 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/saf/xkaoqimfhptjbzdctejwxxhtoqeit, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f885a432-ef61-4df6-a73b-f318a2e16d86, upload has finished false
2024.11.05 19:46:51.124909 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/saf/xkaoqimfhptjbzdctejwxxhtoqeit, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f885a432-ef61-4df6-a73b-f318a2e16d86, upload has finished false
2024.11.05 19:46:51.273334 [ 5795 ] {2d209d86-fdbe-48c9-949a-85bcb01ee050} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=917, jobs=859). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:36146) (comment: 02046_low_cardinality_parallel_group_by.sh) (in query: INSERT INTO low_card_buffer_test values), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&) @ 0x000000001d512c01
10. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:92: DB::CompletedPipelineExecutor::execute() @ 0x000000001d512816
11. /build/src/Server/TCPHandler.cpp:566: DB::TCPHandler::runImpl() @ 0x000000001d4709b1
12. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
13. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
14. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
15. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
16. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
17. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
18. ? @ 0x00007f3c91188ac3
19. ? @ 0x00007f3c9121a850

2024.11.05 19:46:51.275128 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:567: DB::WriteBufferFromS3::writePart(DB::WriteBufferFromS3::PartData&&) @ 0x0000000019914d6e
18. /build/src/IO/WriteBufferFromS3.cpp:370: DB::WriteBufferFromS3::writeMultipartUpload() @ 0x0000000019908eb6
19. /build/src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::nextImpl() @ 0x000000001990857e
20. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
21. /build/src/IO/WriteBufferFromFileDecorator.cpp:71: DB::WriteBufferFromFileDecorator::nextImpl() @ 0x000000001a3593ad
22. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
23. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
24. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
25. /build/src/IO/WriteBuffer.h:76: DB::WriteBuffer::nextIfAtEnd() @ 0x000000000a931978
26. /build/src/IO/WriteBuffer.h:91: DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000a9317e0
27. /build/src/Compression/CompressedWriteBuffer.cpp:58: DB::CompressedWriteBuffer::nextImpl() @ 0x0000000019ddd7ed
28. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
29. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
30. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
31. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:350: DB::MergeTreeDataPartWriterWide::getCurrentMarksForColumn(DB::NameAndTypePair const&, std::set<String, std::less<String>, std::allocator<String>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ccf5839
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:51.277815 [ 5795 ] {2d209d86-fdbe-48c9-949a-85bcb01ee050} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=917, jobs=859). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&) @ 0x000000001d512c01
10. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:92: DB::CompletedPipelineExecutor::execute() @ 0x000000001d512816
11. /build/src/Server/TCPHandler.cpp:566: DB::TCPHandler::runImpl() @ 0x000000001d4709b1
12. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
13. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
14. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
15. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
16. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
17. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
18. ? @ 0x00007f3c91188ac3
19. ? @ 0x00007f3c9121a850

2024.11.05 19:46:51.273350 [ 5796 ] {16131e59-57c0-4158-a58f-b4245f14b193} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=917, jobs=859). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:36154) (comment: 02046_low_cardinality_parallel_group_by.sh) (in query: INSERT INTO low_card_buffer_test values), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&) @ 0x000000001d512c01
10. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:92: DB::CompletedPipelineExecutor::execute() @ 0x000000001d512816
11. /build/src/Server/TCPHandler.cpp:566: DB::TCPHandler::runImpl() @ 0x000000001d4709b1
12. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
13. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
14. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
15. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
16. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
17. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
18. ? @ 0x00007f3c91188ac3
19. ? @ 0x00007f3c9121a850

2024.11.05 19:46:51.282426 [ 5796 ] {16131e59-57c0-4158-a58f-b4245f14b193} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=917, jobs=859). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&) @ 0x000000001d512c01
10. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:92: DB::CompletedPipelineExecutor::execute() @ 0x000000001d512816
11. /build/src/Server/TCPHandler.cpp:566: DB::TCPHandler::runImpl() @ 0x000000001d4709b1
12. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
13. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
14. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
15. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
16. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
17. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
18. ? @ 0x00007f3c91188ac3
19. ? @ 0x00007f3c9121a850

2024.11.05 19:46:51.799430 [ 4766 ] {3cbc29da-7813-4403-89e4-df88facd5e86} <Error> executeQuery: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:35552). (NETWORK_ERROR) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35552) (comment: 00016_any_if_distributed_cond_always_false.sql) (in query: SELECT anyIf(`__table1`.`SearchPhrase`, `__table1`.`CounterID` = -1) AS `anyIf(SearchPhrase, equals(CounterID, -1))` FROM `test`.`hits` AS `__table1`), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:169: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e9244
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Server/TCPHandler.cpp:2217: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001d47db8f
9. /build/src/Server/TCPHandler.cpp:1084: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b9aa
10. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:46:51.919092 [ 4766 ] {3cbc29da-7813-4403-89e4-df88facd5e86} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:35552). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:169: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e9244
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Server/TCPHandler.cpp:2253: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001d487855
9. /build/src/Server/TCPHandler.cpp:2330: DB::TCPHandler::sendLogs() @ 0x000000001d47c5ff
10. /build/src/Server/TCPHandler.cpp:693: DB::TCPHandler::runImpl() @ 0x000000001d471ac2
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:51.921141 [ 4766 ] {3cbc29da-7813-4403-89e4-df88facd5e86} <Error> TCPHandler: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:35552). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:169: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e9244
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Server/TCPHandler.cpp:2217: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001d47db8f
9. /build/src/Server/TCPHandler.cpp:1084: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b9aa
10. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:46:51.924173 [ 4766 ] {3cbc29da-7813-4403-89e4-df88facd5e86} <Warning> TCPHandler: Client has gone away.
2024.11.05 19:46:52.396074 [ 2927 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:57.636631 [ 2933 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:58.441302 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/kam/shyluoqjbvacxkizuogkxewpcuqzr, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b6de99f4-9d2e-4aa4-a805-e8fc2c323f90, upload has finished false.
2024.11.05 19:46:58.442068 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/kam/shyluoqjbvacxkizuogkxewpcuqzr, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b6de99f4-9d2e-4aa4-a805-e8fc2c323f90, upload has finished false
2024.11.05 19:46:58.470395 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/kam/shyluoqjbvacxkizuogkxewpcuqzr, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b6de99f4-9d2e-4aa4-a805-e8fc2c323f90, upload has finished false
2024.11.05 19:46:58.549082 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 15 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:59.239487 [ 4766 ] {ed036432-9ac3-4289-8db3-4472dc3c6086} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:44382) (comment: 00062_loyalty.sql) (in query: SELECT loyalty, count() AS c, bar(log(c + 1) * 1000, 0, log(6000) * 1000, 80) FROM (SELECT UserID, toInt8((yandex > google ? yandex / (yandex + google) : -google / (yandex + google)) * 10) AS loyalty FROM (SELECT UserID, sum(SearchEngineID = 2) AS yandex, sum(SearchEngineID = 3) AS google FROM test.hits WHERE SearchEngineID = 2 OR SearchEngineID = 3 GROUP BY UserID HAVING yandex + google > 10)) GROUP BY loyalty ORDER BY loyalty), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:46:59.466887 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:46:59.496037 [ 2816 ] {faa5185e-b261-461c-8811-ab167b68b393} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:34158) (comment: 00016_any_if_distributed_cond_always_false.sql) (in query: SELECT anyIf(`__table1`.`SearchPhrase`, `__table1`.`CounterID` = -1) AS `anyIf(SearchPhrase, equals(CounterID, -1))` FROM `test`.`hits` AS `__table1`), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:46:59.952778 [ 4766 ] {ed036432-9ac3-4289-8db3-4472dc3c6086} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:47:00.154599 [ 2816 ] {faa5185e-b261-461c-8811-ab167b68b393} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:34158). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:169: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e9244
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Server/TCPHandler.cpp:2253: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001d487855
9. /build/src/Server/TCPHandler.cpp:2330: DB::TCPHandler::sendLogs() @ 0x000000001d47c5ff
10. /build/src/Server/TCPHandler.cpp:693: DB::TCPHandler::runImpl() @ 0x000000001d471ac2
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:00.177154 [ 2816 ] {faa5185e-b261-461c-8811-ab167b68b393} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:47:00.200593 [ 2816 ] {faa5185e-b261-461c-8811-ab167b68b393} <Warning> TCPHandler: Client has gone away.
2024.11.05 19:47:00.705539 [ 3609 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Removing temporary directory /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:47:00.706115 [ 3609 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:47:00.778320 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 2 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:00.806937 [ 2932 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:05.215773 [ 2940 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:06.711733 [ 3810 ] {370ce49a-93cd-470a-8904-39cabab03373} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=22, jobs=22). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:45716) (comment: 01475_read_subcolumns_3.sql) (in query: SELECT count() FROM null_subcolumns WHERE n.null;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:47:06.764793 [ 3810 ] {370ce49a-93cd-470a-8904-39cabab03373} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=22, jobs=22). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:47:07.495569 [ 2941 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:08.126972 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/skt/ywkcdudxqimspilxzvqbsmkwxdwmb, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 064d644d-c3fd-4dda-983e-07cec0b4b068, upload has finished false.
2024.11.05 19:47:08.127896 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/skt/ywkcdudxqimspilxzvqbsmkwxdwmb, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 064d644d-c3fd-4dda-983e-07cec0b4b068, upload has finished false
2024.11.05 19:47:08.184290 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/skt/ywkcdudxqimspilxzvqbsmkwxdwmb, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 064d644d-c3fd-4dda-983e-07cec0b4b068, upload has finished false
2024.11.05 19:47:08.245421 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=14): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 16 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:10.456306 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:12.674238 [ 2929 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:14.729125 [ 2933 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:15.737807 [ 2935 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:15.875854 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/vwe/vvreiehxanszzjvhwrkbnopblclno, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ff507a3f-563e-43cb-a6d4-5f910933c489, upload has finished false.
2024.11.05 19:47:15.876522 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/vwe/vvreiehxanszzjvhwrkbnopblclno, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ff507a3f-563e-43cb-a6d4-5f910933c489, upload has finished false
2024.11.05 19:47:15.916805 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/vwe/vvreiehxanszzjvhwrkbnopblclno, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ff507a3f-563e-43cb-a6d4-5f910933c489, upload has finished false
2024.11.05 19:47:15.986095 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:567: DB::WriteBufferFromS3::writePart(DB::WriteBufferFromS3::PartData&&) @ 0x0000000019914d6e
18. /build/src/IO/WriteBufferFromS3.cpp:370: DB::WriteBufferFromS3::writeMultipartUpload() @ 0x0000000019908eb6
19. /build/src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::nextImpl() @ 0x000000001990857e
20. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
21. /build/src/IO/WriteBufferFromFileDecorator.cpp:71: DB::WriteBufferFromFileDecorator::nextImpl() @ 0x000000001a3593ad
22. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
23. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
24. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
25. /build/src/IO/WriteBuffer.h:76: DB::WriteBuffer::nextIfAtEnd() @ 0x000000000a931978
26. /build/src/IO/WriteBuffer.h:91: DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000a9317e0
27. /build/src/Compression/CompressedWriteBuffer.cpp:58: DB::CompressedWriteBuffer::nextImpl() @ 0x0000000019ddd7ed
28. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
29. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
30. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
31. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:350: DB::MergeTreeDataPartWriterWide::getCurrentMarksForColumn(DB::NameAndTypePair const&, std::set<String, std::less<String>, std::allocator<String>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ccf5839
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:18.164424 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0e641ae6-a90e-4cec-8ebd-a21f62e7934a::202411_1_56_11}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:18.244804 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:18.835029 [ 2937 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:19.124640 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:19.860911 [ 2936 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:20.809500 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:20.997423 [ 2938 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:22.078958 [ 2940 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:27.216967 [ 2928 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:30.338836 [ 2931 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:31.047879 [ 3570 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key s3/sld/cnujeebkvtcazabqpkaidawgatyoc, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 30c7130d-528d-4d9a-9914-b504421f7a7b, upload has finished false.
2024.11.05 19:47:31.048669 [ 3570 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key s3/sld/cnujeebkvtcazabqpkaidawgatyoc, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 30c7130d-528d-4d9a-9914-b504421f7a7b, upload has finished false
2024.11.05 19:47:31.100296 [ 3570 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key s3/sld/cnujeebkvtcazabqpkaidawgatyoc, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 30c7130d-528d-4d9a-9914-b504421f7a7b, upload has finished false
2024.11.05 19:47:31.353407 [ 5793 ] {5c01868b-8b71-4b23-ad05-656818a8a0dc} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=13, jobs=13). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:39748) (comment: 02241_filesystem_cache_on_write_operations.sh) (in query: SELECT count(), sum(size) FROM system.filesystem_cache), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:47:31.355059 [ 2933 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:31.390061 [ 5793 ] {5c01868b-8b71-4b23-ad05-656818a8a0dc} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=13, jobs=13). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:47:31.591945 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=1): (while reading column Referer): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_4_4_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 29 with max_rows_to_read = 6741): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:33.703864 [ 3583 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/vtn/tyvccfyhetoemniduqqhnttofpnfa, total size 234881024, count 234881024, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id dec51c58-ade5-434e-9da2-7b61209160c9, upload has finished false.
2024.11.05 19:47:33.715476 [ 3583 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/vtn/tyvccfyhetoemniduqqhnttofpnfa, total size 234881024, count 234881024, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id dec51c58-ade5-434e-9da2-7b61209160c9, upload has finished false
2024.11.05 19:47:33.792269 [ 3583 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/vtn/tyvccfyhetoemniduqqhnttofpnfa, total size 234881024, count 234881024, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id dec51c58-ade5-434e-9da2-7b61209160c9, upload has finished false
2024.11.05 19:47:33.819467 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:567: DB::WriteBufferFromS3::writePart(DB::WriteBufferFromS3::PartData&&) @ 0x0000000019914d6e
18. /build/src/IO/WriteBufferFromS3.cpp:370: DB::WriteBufferFromS3::writeMultipartUpload() @ 0x0000000019908eb6
19. /build/src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::nextImpl() @ 0x000000001990857e
20. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
21. /build/src/IO/WriteBufferFromFileDecorator.cpp:71: DB::WriteBufferFromFileDecorator::nextImpl() @ 0x000000001a3593ad
22. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
23. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
24. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
25. /build/src/IO/WriteBuffer.h:76: DB::WriteBuffer::nextIfAtEnd() @ 0x000000000a931978
26. /build/src/IO/WriteBuffer.h:91: DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000a9317e0
27. /build/src/Compression/CompressedWriteBuffer.cpp:58: DB::CompressedWriteBuffer::nextImpl() @ 0x0000000019ddd7ed
28. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
29. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
30. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
31. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:350: DB::MergeTreeDataPartWriterWide::getCurrentMarksForColumn(DB::NameAndTypePair const&, std::set<String, std::less<String>, std::allocator<String>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ccf5839
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:34.432391 [ 2932 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:34.746902 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:34.918176 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:36.192347 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:36.358936 [ 5793 ] {75351330-9810-49bc-864f-8cbc5fa25954} <Error> executeQuery: Code: 70. DB::Exception: Conversion from AggregateFunction(sum, UInt8) to AggregateFunction(sum, Int32) is not supported: while converting source column pv to destination column pv. (CANNOT_CONVERT_TYPE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:56020) (comment: 01880_materialized_view_to_table_type_check.sql) (in query: CREATE MATERIALIZED VIEW test_mv to test(`id` Int32, `pv` AggregateFunction(sum, Int32)) as SELECT id, sumState(1) as pv from test_input group by id;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000a8f2fcc
6. /build/src/Functions/FunctionsConversion.cpp:3480: DB::(anonymous namespace)::FunctionCast::createAggregateFunctionWrapper(std::shared_ptr<DB::IDataType const> const&, DB::DataTypeAggregateFunction const*) const @ 0x000000000aa7276f
7. /build/src/Functions/FunctionsConversion.cpp:4779: DB::(anonymous namespace)::FunctionCast::prepareImpl(std::shared_ptr<DB::IDataType const> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000000aa700fd
8. /build/src/Functions/FunctionsConversion.cpp:4495: DB::(anonymous namespace)::FunctionCast::prepareRemoveNullable(std::shared_ptr<DB::IDataType const> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000000aa6afbc
9. /build/src/Functions/FunctionsConversion.cpp:4423: DB::(anonymous namespace)::FunctionCast::prepareUnpackDictionaries(std::shared_ptr<DB::IDataType const> const&, std::shared_ptr<DB::IDataType const> const&) const @ 0x000000000aa6a270
10. /build/src/Functions/FunctionsConversion.cpp:3106: DB::(anonymous namespace)::FunctionCast::prepare(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000aa69adc
11. /build/src/Interpreters/ActionsDAG.cpp:317: DB::ActionsDAG::addFunctionImpl(std::shared_ptr<DB::IFunctionBase const> const&, std::vector<DB::ActionsDAG::Node const*, std::allocator<DB::ActionsDAG::Node const*>>, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>, String, std::shared_ptr<DB::IDataType const>, bool) @ 0x000000001a437798
12. /build/src/Interpreters/ActionsDAG.cpp:244: DB::ActionsDAG::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<DB::ActionsDAG::Node const*, std::allocator<DB::ActionsDAG::Node const*>>, String) @ 0x000000001a43715b
13. /build/src/Interpreters/ActionsDAG.cpp:1455: DB::ActionsDAG::makeConvertingActions(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, DB::ActionsDAG::MatchColumnsMode, bool, bool, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>*) @ 0x000000001a4417a3
14. /build/src/Interpreters/InterpreterCreateQuery.cpp:1257: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b017eeb
15. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
16. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
17. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
18. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
19. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
20. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
21. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
22. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
23. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
24. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
25. ? @ 0x00007f3c91188ac3
26. ? @ 0x00007f3c9121a850

2024.11.05 19:47:36.369222 [ 5793 ] {75351330-9810-49bc-864f-8cbc5fa25954} <Error> TCPHandler: Code: 70. DB::Exception: Conversion from AggregateFunction(sum, UInt8) to AggregateFunction(sum, Int32) is not supported: while converting source column pv to destination column pv. (CANNOT_CONVERT_TYPE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000a8f2fcc
6. /build/src/Functions/FunctionsConversion.cpp:3480: DB::(anonymous namespace)::FunctionCast::createAggregateFunctionWrapper(std::shared_ptr<DB::IDataType const> const&, DB::DataTypeAggregateFunction const*) const @ 0x000000000aa7276f
7. /build/src/Functions/FunctionsConversion.cpp:4779: DB::(anonymous namespace)::FunctionCast::prepareImpl(std::shared_ptr<DB::IDataType const> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000000aa700fd
8. /build/src/Functions/FunctionsConversion.cpp:4495: DB::(anonymous namespace)::FunctionCast::prepareRemoveNullable(std::shared_ptr<DB::IDataType const> const&, std::shared_ptr<DB::IDataType const> const&, bool) const @ 0x000000000aa6afbc
9. /build/src/Functions/FunctionsConversion.cpp:4423: DB::(anonymous namespace)::FunctionCast::prepareUnpackDictionaries(std::shared_ptr<DB::IDataType const> const&, std::shared_ptr<DB::IDataType const> const&) const @ 0x000000000aa6a270
10. /build/src/Functions/FunctionsConversion.cpp:3106: DB::(anonymous namespace)::FunctionCast::prepare(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000aa69adc
11. /build/src/Interpreters/ActionsDAG.cpp:317: DB::ActionsDAG::addFunctionImpl(std::shared_ptr<DB::IFunctionBase const> const&, std::vector<DB::ActionsDAG::Node const*, std::allocator<DB::ActionsDAG::Node const*>>, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>, String, std::shared_ptr<DB::IDataType const>, bool) @ 0x000000001a437798
12. /build/src/Interpreters/ActionsDAG.cpp:244: DB::ActionsDAG::addFunction(std::shared_ptr<DB::IFunctionOverloadResolver> const&, std::vector<DB::ActionsDAG::Node const*, std::allocator<DB::ActionsDAG::Node const*>>, String) @ 0x000000001a43715b
13. /build/src/Interpreters/ActionsDAG.cpp:1455: DB::ActionsDAG::makeConvertingActions(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, DB::ActionsDAG::MatchColumnsMode, bool, bool, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>*) @ 0x000000001a4417a3
14. /build/src/Interpreters/InterpreterCreateQuery.cpp:1257: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b017eeb
15. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
16. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
17. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
18. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
19. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
20. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
21. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
22. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
23. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
24. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
25. ? @ 0x00007f3c91188ac3
26. ? @ 0x00007f3c9121a850

2024.11.05 19:47:37.047876 [ 5793 ] {b48400eb-089a-4ac4-be6b-cd9d595d58f3} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=13, jobs=13). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:56020) (comment: 01880_materialized_view_to_table_type_check.sql) (in query: INSERT INTO test_input SELECT toInt32(number % 1000) AS id FROM numbers(10);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:47:37.126642 [ 5793 ] {b48400eb-089a-4ac4-be6b-cd9d595d58f3} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=13, jobs=13). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:47:37.547854 [ 5215 ] {5417b94c-7183-4a36-b32a-3bb3e1a747c5} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=24, jobs=24). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46278) (comment: 00024_random_counters.sql) (in query: SELECT uniq(UserID), sum(Sign) FROM test.visits WHERE CounterID = 9627212;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:47:37.661108 [ 5215 ] {5417b94c-7183-4a36-b32a-3bb3e1a747c5} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=24, jobs=24). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:47:41.594442 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/bes/yxyvzidhmiqsclneiwlolsryielnt, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id e0d84087-6cc2-4833-95c9-f5ba306c1da9, upload has finished false.
2024.11.05 19:47:41.604772 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/bes/yxyvzidhmiqsclneiwlolsryielnt, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id e0d84087-6cc2-4833-95c9-f5ba306c1da9, upload has finished false
2024.11.05 19:47:41.646262 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/bes/yxyvzidhmiqsclneiwlolsryielnt, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id e0d84087-6cc2-4833-95c9-f5ba306c1da9, upload has finished false
2024.11.05 19:47:41.721374 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 16 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:44.988951 [ 2930 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:47.412666 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:48.674682 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/vrw/yqjcnuamggrvezokipyilxvvutrbq, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ead952b3-b270-498d-b24f-9ade1c9e65d3, upload has finished false.
2024.11.05 19:47:48.687722 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/vrw/yqjcnuamggrvezokipyilxvvutrbq, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ead952b3-b270-498d-b24f-9ade1c9e65d3, upload has finished false
2024.11.05 19:47:48.734225 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/vrw/yqjcnuamggrvezokipyilxvvutrbq, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ead952b3-b270-498d-b24f-9ade1c9e65d3, upload has finished false
2024.11.05 19:47:48.854112 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 17 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:49.479285 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:50.198895 [ 2934 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:50.466544 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:51.238400 [ 5795 ] {f17faa92-f6cf-4162-b899-da036a9ac24e} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=26, jobs=26). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:56028) (comment: 00625_arrays_in_nested.sql) (in query: SELECT * FROM nested;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:47:51.272168 [ 5795 ] {f17faa92-f6cf-4162-b899-da036a9ac24e} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=26, jobs=26). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:47:52.158320 [ 5796 ] {3cc792c7-22f3-4503-9105-d53b0d1c6c28} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=21, jobs=21). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:59964) (comment: 00082_quantiles.sql) (in query: SELECT CounterID AS k, quantileExact(0.5)(ResolutionWidth) FROM test.hits GROUP BY k ORDER BY count() DESC, CounterID LIMIT 10;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:47:52.238676 [ 2937 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:52.460577 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:274: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3abc4
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:47:54.207869 [ 5796 ] {3cc792c7-22f3-4503-9105-d53b0d1c6c28} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=21, jobs=21). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:47:59.612768 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/tiv/ochuzxhscdsqytjkrotrxhgingyds, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 2033e33a-1246-45ef-80cc-f5735a94a725, upload has finished false.
2024.11.05 19:47:59.617855 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/tiv/ochuzxhscdsqytjkrotrxhgingyds, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 2033e33a-1246-45ef-80cc-f5735a94a725, upload has finished false
2024.11.05 19:47:59.663476 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/tiv/ochuzxhscdsqytjkrotrxhgingyds, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 2033e33a-1246-45ef-80cc-f5735a94a725, upload has finished false
2024.11.05 19:47:59.749026 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 25 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:01.120383 [ 5793 ] {d611f0ee-578e-4230-8029-ce695f671efd} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=23, jobs=23). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:55084) (comment: 00097_constexpr_in_index.sql) (in query: -- Even in presence of OR, we evaluate the "0 IN (1, 2, 3)" as a constant expression therefore it does not prevent the index analysis.
 SELECT count() FROM test.hits WHERE CounterID IN (14917930, 33034174) OR 0 IN (1, 2, 3) SETTINGS max_rows_to_read = 1000000, force_primary_key = 1;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:48:01.239717 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {dbd666d3-2d23-40c6-8136-6ca2be336b4d::202411_1_51_9}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:01.243569 [ 5793 ] {d611f0ee-578e-4230-8029-ce695f671efd} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=23, jobs=23). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:48:02.588254 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:03.560443 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 8 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:05.350869 [ 5795 ] {96a98c38-670b-40a9-b609-f6ab26afcfaa} <Error> executeQuery: Code: 407. DB::Exception: Decimal math overflow: In scope SELECT arrayDifference([toDecimal32(100.0000991821289, 0), -2147483647]) AS x. (DECIMAL_OVERFLOW) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:58862) (comment: 01851_array_difference_decimal_overflow_ubsan.sql) (in query: SELECT arrayDifference([toDecimal32(100.0000991821289, 0), -2147483647]) AS x;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Functions/array/arrayDifference.cpp:91: void DB::ArrayDifferenceImpl::impl<DB::Decimal<int>, DB::Decimal<int>>(DB::Decimal<int> const*, DB::Decimal<int>*, unsigned long, unsigned long) @ 0x00000000121bcea8
7. /build/src/Functions/array/arrayDifference.cpp:132: bool DB::ArrayDifferenceImpl::executeType<DB::Decimal<int>, DB::Decimal<int>>(COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&, DB::ColumnArray const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn>&) @ 0x00000000121bafc6
8. /build/src/Functions/array/arrayDifference.cpp:153: DB::ArrayDifferenceImpl::execute(DB::ColumnArray const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn>) @ 0x00000000121b7eb9
9. /build/src/Functions/array/FunctionArrayMapped.h:278: DB::FunctionArrayMapped<DB::ArrayDifferenceImpl, DB::NameArrayDifference>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000121b4588
10. /build/src/Functions/IFunction.h:443: DB::IFunction::executeImplDryRun(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000a8efa99
11. /build/src/Functions/IFunctionAdaptors.h:26: DB::FunctionToExecutableFunctionAdaptor::executeDryRunImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000af6b1ad
12. /build/src/Functions/IFunction.cpp:277: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062f06
13. /build/src/Functions/IFunction.cpp:191: DB::IExecutableFunction::defaultImplementationForConstantArguments(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062a83
14. /build/src/Functions/IFunction.cpp:269: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062de3
15. /build/src/Functions/IFunction.cpp:336: DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019064026
16. /build/src/Functions/IFunction.cpp:411: DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000190650b5
17. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6185: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be389
18. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
19. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221
20. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6901: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0ab9c7
21. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8173: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d543
22. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
23. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
24. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
25. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
26. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
27. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
28. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d

2024.11.05 19:48:05.361487 [ 5795 ] {96a98c38-670b-40a9-b609-f6ab26afcfaa} <Error> TCPHandler: Code: 407. DB::Exception: Decimal math overflow: In scope SELECT arrayDifference([toDecimal32(100.0000991821289, 0), -2147483647]) AS x. (DECIMAL_OVERFLOW), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Functions/array/arrayDifference.cpp:91: void DB::ArrayDifferenceImpl::impl<DB::Decimal<int>, DB::Decimal<int>>(DB::Decimal<int> const*, DB::Decimal<int>*, unsigned long, unsigned long) @ 0x00000000121bcea8
7. /build/src/Functions/array/arrayDifference.cpp:132: bool DB::ArrayDifferenceImpl::executeType<DB::Decimal<int>, DB::Decimal<int>>(COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&, DB::ColumnArray const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn>&) @ 0x00000000121bafc6
8. /build/src/Functions/array/arrayDifference.cpp:153: DB::ArrayDifferenceImpl::execute(DB::ColumnArray const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn>) @ 0x00000000121b7eb9
9. /build/src/Functions/array/FunctionArrayMapped.h:278: DB::FunctionArrayMapped<DB::ArrayDifferenceImpl, DB::NameArrayDifference>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x00000000121b4588
10. /build/src/Functions/IFunction.h:443: DB::IFunction::executeImplDryRun(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000a8efa99
11. /build/src/Functions/IFunctionAdaptors.h:26: DB::FunctionToExecutableFunctionAdaptor::executeDryRunImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000af6b1ad
12. /build/src/Functions/IFunction.cpp:277: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062f06
13. /build/src/Functions/IFunction.cpp:191: DB::IExecutableFunction::defaultImplementationForConstantArguments(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062a83
14. /build/src/Functions/IFunction.cpp:269: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062de3
15. /build/src/Functions/IFunction.cpp:336: DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019064026
16. /build/src/Functions/IFunction.cpp:411: DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000190650b5
17. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6185: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be389
18. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
19. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221
20. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6901: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0ab9c7
21. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8173: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d543
22. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
23. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
24. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
25. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
26. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
27. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
28. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d

2024.11.05 19:48:05.600010 [ 2935 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:06.624417 [ 2934 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:06.815025 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=14, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:09.410265 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/rwm/ycmcqdsrdvpllyaptelyosobkkhmv, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 5160f171-f922-40b9-9d8a-a850a56734f6, upload has finished false.
2024.11.05 19:48:09.411905 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/rwm/ycmcqdsrdvpllyaptelyosobkkhmv, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 5160f171-f922-40b9-9d8a-a850a56734f6, upload has finished false
2024.11.05 19:48:09.463353 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/rwm/ycmcqdsrdvpllyaptelyosobkkhmv, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 5160f171-f922-40b9-9d8a-a850a56734f6, upload has finished false
2024.11.05 19:48:09.556653 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 13 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:10.824903 [ 5798 ] {11c34a3f-d610-460d-b4a9-2adc517a60f2} <Error> executeQuery: Code: 57. DB::Exception: Table test_1.test_table already exists. (TABLE_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:58888) (comment: 02517_executable_pool_bad_input_query.sql) (in query: CREATE TABLE test_table (value String) ENGINE=ExecutablePool('nonexist.py', 'TabSeparated', (foobar));), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000a8f2fcc
6. /build/src/Interpreters/InterpreterCreateQuery.cpp:1395: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x000000001b01c684
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:1313: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b018b66
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
9. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
10. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
11. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
12. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
13. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
14. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
15. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
16. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
17. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
18. ? @ 0x00007f3c91188ac3
19. ? @ 0x00007f3c9121a850

2024.11.05 19:48:10.828608 [ 5798 ] {11c34a3f-d610-460d-b4a9-2adc517a60f2} <Error> TCPHandler: Code: 57. DB::Exception: Table test_1.test_table already exists. (TABLE_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000a8f2fcc
6. /build/src/Interpreters/InterpreterCreateQuery.cpp:1395: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x000000001b01c684
7. /build/src/Interpreters/InterpreterCreateQuery.cpp:1313: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b018b66
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
9. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
10. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
11. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
12. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
13. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
14. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
15. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
16. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
17. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
18. ? @ 0x00007f3c91188ac3
19. ? @ 0x00007f3c9121a850

2024.11.05 19:48:12.405234 [ 5796 ] {dfeca41a-9235-4d39-8cd8-cfb263af38c1} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=21, jobs=20). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:58884) (comment: 00384_column_aggregate_function_insert_from.sql) (in query: INSERT INTO aggregates SELECT toDate('2016-10-31') AS d, uniqState(toUInt64(arrayJoin(range(100)))) AS s UNION ALL SELECT toDate('2016-11-01') AS d, uniqState(toUInt64(arrayJoin(range(100)))) AS s;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:48:12.845493 [ 5796 ] {dfeca41a-9235-4d39-8cd8-cfb263af38c1} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=21, jobs=20). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:48:12.874275 [ 2939 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:15.935314 [ 2927 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:17.892967 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_66_13}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:17.995923 [ 2930 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:19.006367 [ 2929 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:19.816493 [ 3610 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Removing temporary directory /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:48:19.817229 [ 3610 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:48:19.915540 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/aox/szlzevnryyylkcevijatuutsfsdtt, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ba7ae046-b77b-4490-8582-b037824ca6a2, upload has finished false.
2024.11.05 19:48:19.916799 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/aox/szlzevnryyylkcevijatuutsfsdtt, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ba7ae046-b77b-4490-8582-b037824ca6a2, upload has finished false
2024.11.05 19:48:19.959523 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/aox/szlzevnryyylkcevijatuutsfsdtt, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ba7ae046-b77b-4490-8582-b037824ca6a2, upload has finished false
2024.11.05 19:48:19.975765 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 28 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:23.490119 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 6 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:25.237998 [ 2937 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:27.313196 [ 2938 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:28.324133 [ 2940 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:28.397854 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 7 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:29.337355 [ 2939 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:30.373269 [ 2941 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:30.410555 [ 5215 ] {7fde67cb-1b63-4ed9-b785-0165de89349f} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=17, jobs=17). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:59106) (comment: 02314_avro_null_as_default.sh) (in query: select * from file(data_02314.avro, auto, 'x UInt32') settings input_format_avro_null_as_default=1), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:48:30.451756 [ 5215 ] {7fde67cb-1b63-4ed9-b785-0165de89349f} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=17, jobs=17). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:48:30.881826 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:31.390183 [ 2926 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:33.307283 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:33.905664 [ 5798 ] {9e554133-2166-4c92-ab3b-aafee4ba18dd} <Error> executeQuery: Code: 43. DB::Exception: Illegal types Array(UInt8) and Array(UInt8) of arguments of function modulo: In scope SELECT [12, 13] % [5, 6]. (ILLEGAL_TYPE_OF_ARGUMENT) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:59088) (comment: 02812_pointwise_array_operations.sql) (in query: SELECT ([12,13] % [5,6]);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000aa280a8
6. /build/src/Functions/FunctionBinaryArithmetic.h:1753: DB::FunctionBinaryArithmetic<DB::ModuloImpl, DB::NameModulo, false, true, false>::getReturnTypeImplStatic(std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, std::shared_ptr<DB::Context const>) @ 0x0000000010125ed2
7. /build/src/Functions/FunctionBinaryArithmetic.h:2664: DB::BinaryArithmeticOverloadResolver<DB::ModuloImpl, DB::NameModulo, false, true>::getReturnTypeImpl(std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&) const @ 0x000000000fe8e04f
8. /build/src/Functions/IFunction.h:386: DB::IFunctionOverloadResolver::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000b45b929
9. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
10. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
11. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
12. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
13. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
14. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221
15. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6901: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0ab9c7
16. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8173: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d543
17. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
18. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
19. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
20. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
21. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
22. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
23. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
30. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
31. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0

2024.11.05 19:48:33.908901 [ 5798 ] {9e554133-2166-4c92-ab3b-aafee4ba18dd} <Error> TCPHandler: Code: 43. DB::Exception: Illegal types Array(UInt8) and Array(UInt8) of arguments of function modulo: In scope SELECT [12, 13] % [5, 6]. (ILLEGAL_TYPE_OF_ARGUMENT), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000aa280a8
6. /build/src/Functions/FunctionBinaryArithmetic.h:1753: DB::FunctionBinaryArithmetic<DB::ModuloImpl, DB::NameModulo, false, true, false>::getReturnTypeImplStatic(std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, std::shared_ptr<DB::Context const>) @ 0x0000000010125ed2
7. /build/src/Functions/FunctionBinaryArithmetic.h:2664: DB::BinaryArithmeticOverloadResolver<DB::ModuloImpl, DB::NameModulo, false, true>::getReturnTypeImpl(std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&) const @ 0x000000000fe8e04f
8. /build/src/Functions/IFunction.h:386: DB::IFunctionOverloadResolver::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000b45b929
9. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
10. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
11. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
12. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
13. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
14. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221
15. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6901: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0ab9c7
16. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8173: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d543
17. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
18. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
19. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
20. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
21. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
22. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
23. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
30. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
31. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0

2024.11.05 19:48:34.249155 [ 5798 ] {92b632c7-f172-4c45-9e3b-55fa74a86da1} <Error> executeQuery: Code: 190. DB::Exception: Two arguments for function minus must have equal sizes: In scope SELECT [2, 3, 4] - [1, -2, 10, 29]. (SIZES_OF_ARRAYS_DONT_MATCH) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:59088) (comment: 02812_pointwise_array_operations.sql) (in query: SELECT ([2,3,4]-[1,-2,10,29]);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Functions/FunctionBinaryArithmetic.h:1226: DB::FunctionBinaryArithmetic<DB::MinusImpl, DB::NameMinus, true, true, false>::executeArraysImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000fc4b90c
7. /build/src/Functions/FunctionBinaryArithmetic.h:2322: DB::FunctionBinaryArithmetic<DB::MinusImpl, DB::NameMinus, true, true, false>::executeImpl2(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const*) const @ 0x000000000fc49f9b
8. /build/src/Functions/FunctionBinaryArithmetic.h:0: DB::FunctionBinaryArithmetic<DB::MinusImpl, DB::NameMinus, true, true, false>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000fc46a36
9. /build/src/Functions/FunctionBinaryArithmetic.h:0: DB::FunctionBinaryArithmeticWithConstants<DB::MinusImpl, DB::NameMinus, true, true, false>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000fc4562c
10. /build/src/Functions/FunctionBinaryArithmetic.h:1211: DB::FunctionBinaryArithmetic<DB::MinusImpl, DB::NameMinus, true, true, false>::executeArraysImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000fc4b50f
11. /build/src/Functions/FunctionBinaryArithmetic.h:2322: DB::FunctionBinaryArithmetic<DB::MinusImpl, DB::NameMinus, true, true, false>::executeImpl2(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const*) const @ 0x000000000fc49f9b
12. /build/src/Functions/FunctionBinaryArithmetic.h:0: DB::FunctionBinaryArithmetic<DB::MinusImpl, DB::NameMinus, true, true, false>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000fc46a36
13. /build/src/Functions/FunctionBinaryArithmetic.h:0: DB::FunctionBinaryArithmeticWithConstants<DB::MinusImpl, DB::NameMinus, true, true, false>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000fc4562c
14. /build/src/Functions/IFunction.h:443: DB::IFunction::executeImplDryRun(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000a8efa99
15. /build/src/Functions/IFunctionAdaptors.h:26: DB::FunctionToExecutableFunctionAdaptor::executeDryRunImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000af6b1ad
16. /build/src/Functions/IFunction.cpp:277: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062f06
17. /build/src/Functions/IFunction.cpp:336: DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019064026
18. /build/src/Functions/IFunction.cpp:411: DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000190650b5
19. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6185: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be389
20. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
21. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221
22. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6901: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0ab9c7
23. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8173: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d543
24. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
25. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
26. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
27. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
28. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
29. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
30. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd

2024.11.05 19:48:34.271943 [ 5798 ] {92b632c7-f172-4c45-9e3b-55fa74a86da1} <Error> TCPHandler: Code: 190. DB::Exception: Two arguments for function minus must have equal sizes: In scope SELECT [2, 3, 4] - [1, -2, 10, 29]. (SIZES_OF_ARRAYS_DONT_MATCH), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Functions/FunctionBinaryArithmetic.h:1226: DB::FunctionBinaryArithmetic<DB::MinusImpl, DB::NameMinus, true, true, false>::executeArraysImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000fc4b90c
7. /build/src/Functions/FunctionBinaryArithmetic.h:2322: DB::FunctionBinaryArithmetic<DB::MinusImpl, DB::NameMinus, true, true, false>::executeImpl2(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const*) const @ 0x000000000fc49f9b
8. /build/src/Functions/FunctionBinaryArithmetic.h:0: DB::FunctionBinaryArithmetic<DB::MinusImpl, DB::NameMinus, true, true, false>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000fc46a36
9. /build/src/Functions/FunctionBinaryArithmetic.h:0: DB::FunctionBinaryArithmeticWithConstants<DB::MinusImpl, DB::NameMinus, true, true, false>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000fc4562c
10. /build/src/Functions/FunctionBinaryArithmetic.h:1211: DB::FunctionBinaryArithmetic<DB::MinusImpl, DB::NameMinus, true, true, false>::executeArraysImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000fc4b50f
11. /build/src/Functions/FunctionBinaryArithmetic.h:2322: DB::FunctionBinaryArithmetic<DB::MinusImpl, DB::NameMinus, true, true, false>::executeImpl2(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const*) const @ 0x000000000fc49f9b
12. /build/src/Functions/FunctionBinaryArithmetic.h:0: DB::FunctionBinaryArithmetic<DB::MinusImpl, DB::NameMinus, true, true, false>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000fc46a36
13. /build/src/Functions/FunctionBinaryArithmetic.h:0: DB::FunctionBinaryArithmeticWithConstants<DB::MinusImpl, DB::NameMinus, true, true, false>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000fc4562c
14. /build/src/Functions/IFunction.h:443: DB::IFunction::executeImplDryRun(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000a8efa99
15. /build/src/Functions/IFunctionAdaptors.h:26: DB::FunctionToExecutableFunctionAdaptor::executeDryRunImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000af6b1ad
16. /build/src/Functions/IFunction.cpp:277: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062f06
17. /build/src/Functions/IFunction.cpp:336: DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019064026
18. /build/src/Functions/IFunction.cpp:411: DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000190650b5
19. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6185: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be389
20. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
21. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221
22. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6901: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0ab9c7
23. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8173: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d543
24. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
25. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
26. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
27. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
28. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
29. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
30. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd

2024.11.05 19:48:35.220748 [ 5798 ] {237821cc-68a4-4840-b6d2-42c57de3fe5d} <Error> executeQuery: Code: 190. DB::Exception: Two arguments for function plus must have equal sizes: while executing 'FUNCTION plus(__table1.x :: 2, __table1.y : 0) -> plus(__table1.x, __table1.y) Array(UInt64) : 3'. (SIZES_OF_ARRAYS_DONT_MATCH) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:59088) (comment: 02812_pointwise_array_operations.sql) (in query: SELECT x, y, x+y FROM a;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Functions/FunctionBinaryArithmetic.h:1226: DB::FunctionBinaryArithmetic<DB::PlusImpl, DB::NamePlus, true, true, false>::executeArraysImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001116716c
7. /build/src/Functions/FunctionBinaryArithmetic.h:2322: DB::FunctionBinaryArithmetic<DB::PlusImpl, DB::NamePlus, true, true, false>::executeImpl2(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const*) const @ 0x00000000111657fb
8. /build/src/Functions/FunctionBinaryArithmetic.h:0: DB::FunctionBinaryArithmetic<DB::PlusImpl, DB::NamePlus, true, true, false>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x0000000011162216
9. /build/src/Functions/IFunctionAdaptors.h:21: DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000af6b14d
10. /build/src/Functions/IFunction.cpp:279: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062f5a
11. /build/src/Functions/IFunction.cpp:336: DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019064026
12. /build/src/Functions/IFunction.cpp:411: DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000190650b5
13. /build/src/Interpreters/ExpressionActions.cpp:616: DB::executeAction(DB::ExpressionActions::Action const&, DB::(anonymous namespace)::ExecutionContext&, bool, bool) @ 0x000000001a6a26b7
14. /build/src/Interpreters/ExpressionActions.cpp:746: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x000000001a6a1a47
15. /build/src/Processors/Transforms/ExpressionTransform.cpp:23: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000001da42c48
16. /build/src/Processors/ISimpleTransform.h:33: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000016042b62
17. /build/src/Processors/ISimpleTransform.cpp:89: DB::ISimpleTransform::work() @ 0x000000001d4f51d2
18. /build/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001d5343e3
19. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001d534120
20. /build/src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d5166a1
21. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
22. /build/src/Processors/Executors/PipelineExecutor.cpp:411: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d5156d5
23. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
24. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
25. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
27. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
28. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
29. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5

2024.11.05 19:48:35.241334 [ 5798 ] {237821cc-68a4-4840-b6d2-42c57de3fe5d} <Error> TCPHandler: Code: 190. DB::Exception: Two arguments for function plus must have equal sizes: while executing 'FUNCTION plus(__table1.x :: 2, __table1.y : 0) -> plus(__table1.x, __table1.y) Array(UInt64) : 3'. (SIZES_OF_ARRAYS_DONT_MATCH), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Functions/FunctionBinaryArithmetic.h:1226: DB::FunctionBinaryArithmetic<DB::PlusImpl, DB::NamePlus, true, true, false>::executeArraysImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001116716c
7. /build/src/Functions/FunctionBinaryArithmetic.h:2322: DB::FunctionBinaryArithmetic<DB::PlusImpl, DB::NamePlus, true, true, false>::executeImpl2(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const*) const @ 0x00000000111657fb
8. /build/src/Functions/FunctionBinaryArithmetic.h:0: DB::FunctionBinaryArithmetic<DB::PlusImpl, DB::NamePlus, true, true, false>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x0000000011162216
9. /build/src/Functions/IFunctionAdaptors.h:21: DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000af6b14d
10. /build/src/Functions/IFunction.cpp:279: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062f5a
11. /build/src/Functions/IFunction.cpp:336: DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019064026
12. /build/src/Functions/IFunction.cpp:411: DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000190650b5
13. /build/src/Interpreters/ExpressionActions.cpp:616: DB::executeAction(DB::ExpressionActions::Action const&, DB::(anonymous namespace)::ExecutionContext&, bool, bool) @ 0x000000001a6a26b7
14. /build/src/Interpreters/ExpressionActions.cpp:746: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x000000001a6a1a47
15. /build/src/Processors/Transforms/ExpressionTransform.cpp:23: DB::ExpressionTransform::transform(DB::Chunk&) @ 0x000000001da42c48
16. /build/src/Processors/ISimpleTransform.h:33: DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x0000000016042b62
17. /build/src/Processors/ISimpleTransform.cpp:89: DB::ISimpleTransform::work() @ 0x000000001d4f51d2
18. /build/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001d5343e3
19. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001d534120
20. /build/src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d5166a1
21. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
22. /build/src/Processors/Executors/PipelineExecutor.cpp:411: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d5156d5
23. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
24. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
25. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
27. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
28. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
29. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5

2024.11.05 19:48:37.107235 [ 5793 ] {683353d4-825c-4860-aa2c-a219058628ad} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:59046) (comment: 00178_quantile_ddsketch.sql) (in query: SELECT CounterID AS k, round(quantileDD(0.01, 0.5)(ResolutionWidth), 2) FROM test.hits GROUP BY k ORDER BY count() DESC, CounterID LIMIT 10;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:48:37.665746 [ 2933 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:38.262267 [ 5793 ] {683353d4-825c-4860-aa2c-a219058628ad} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:48:38.973744 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column Referer): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_2_2_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 40 with max_rows_to_read = 6592): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:39.461674 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/eoa/flosmvxqjekbtuzpznzllxjvypojl, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f088106e-755f-4434-a270-add43b896ecb, upload has finished false.
2024.11.05 19:48:39.462833 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/eoa/flosmvxqjekbtuzpznzllxjvypojl, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f088106e-755f-4434-a270-add43b896ecb, upload has finished false
2024.11.05 19:48:39.549718 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/eoa/flosmvxqjekbtuzpznzllxjvypojl, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f088106e-755f-4434-a270-add43b896ecb, upload has finished false
2024.11.05 19:48:39.647657 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 31 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:39.832946 [ 2934 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:40.866458 [ 2932 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:41.884847 [ 2937 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:42.407629 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 2 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:43.953023 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=1): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:44.000344 [ 2938 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:44.285787 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:46.500439 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:47.918506 [ 3810 ] {c38bcc90-7ee3-472d-a168-dfdd35588563} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=16, jobs=16). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:42142) (comment: 01913_join_push_down_bug.sql) (in query: SELECT id, flag FROM test t1 INNER JOIN (SELECT DISTINCT id FROM test) AS t2 ON t1.id = t2.id WHERE flag = 0 and t = 1 AND id NOT IN (SELECT 1 WHERE 0) ORDER BY id;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:48:48.752402 [ 3810 ] {c38bcc90-7ee3-472d-a168-dfdd35588563} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=16, jobs=16). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:48:50.296394 [ 2928 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:51.318929 [ 2930 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:51.454523 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 10 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:52.138194 [ 5215 ] {9944d8de-c87e-41a8-b6ed-9928615c500d} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=40, jobs=40). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:42158) (comment: 00040_array_enumerate_uniq.sql) (in query: SELECT max(arrayJoin(arr)) FROM (SELECT arrayEnumerateUniq(groupArray(intDiv(number, 54321)) AS nums, groupArray(toString(intDiv(number, 98765)))) AS arr FROM (SELECT number FROM system.numbers LIMIT 1000000) GROUP BY intHash32(number) % 100000);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:48:52.312665 [ 5215 ] {9944d8de-c87e-41a8-b6ed-9928615c500d} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=40, jobs=40). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:48:52.839561 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:53.793807 [ 3810 ] {a1adaec1-ede3-4668-af95-c9b49d2dcdde} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=4, jobs=4). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:42162) (comment: 00955_test_final_mark.sql) (in query: SELECT COUNT(*) FROM mt_with_pk WHERE x > toDateTime('2018-10-01 23:57:57');), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:48:53.805826 [ 3810 ] {a1adaec1-ede3-4668-af95-c9b49d2dcdde} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=4, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:48:53.976982 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:54.390116 [ 2933 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:57.018009 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 5 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:58.579451 [ 2937 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:59.590461 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:48:59.595351 [ 2936 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:00.383940 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {dbd666d3-2d23-40c6-8136-6ca2be336b4d::202411_1_57_10}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:00.479598 [ 5798 ] {9bd3da69-128b-4a92-a941-f25ad6873c8b} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=30, jobs=30). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49040) (comment: 01572_kill_window_function.sh) (in query: SELECT count() FROM system.processes WHERE query_id = '01572_kill_window_function-test_1'), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:49:00.607436 [ 5798 ] {9bd3da69-128b-4a92-a941-f25ad6873c8b} <Error> DynamicQueryHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=30, jobs=30). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:05.213143 [ 3810 ] {01572_kill_window_function-test_1} <Error> executeQuery: Code: 394. DB::Exception: Query was cancelled. (QUERY_WAS_CANCELLED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:53786) (comment: 01572_kill_window_function.sh) (in query: SELECT sum(number) OVER (PARTITION BY number % 10 ORDER BY number DESC NULLS FIRST ROWS BETWEEN CURRENT ROW AND 99999 FOLLOWING) FROM numbers(0, 10000000) format Null;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Interpreters/ProcessList.cpp:480: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c05
7. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
8. /build/src/Processors/Executors/PipelineExecutor.cpp:196: DB::PipelineExecutor::finalizeExecution() @ 0x000000001d515c7f
9. /build/src/Processors/Executors/PipelineExecutor.cpp:131: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d51553e
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
12. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
13. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
14. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
15. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
22. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
23. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
25. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
26. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
27. ? @ 0x00007f3c91188ac3
28. ? @ 0x00007f3c9121a850

2024.11.05 19:49:05.224158 [ 3810 ] {01572_kill_window_function-test_1} <Error> TCPHandler: Code: 394. DB::Exception: Query was cancelled. (QUERY_WAS_CANCELLED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Interpreters/ProcessList.cpp:480: DB::QueryStatus::checkTimeLimit() @ 0x000000001b6a6c05
7. /build/src/Processors/Executors/PipelineExecutor.cpp:186: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001d5155c0
8. /build/src/Processors/Executors/PipelineExecutor.cpp:196: DB::PipelineExecutor::finalizeExecution() @ 0x000000001d515c7f
9. /build/src/Processors/Executors/PipelineExecutor.cpp:131: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d51553e
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
12. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
13. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
14. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
15. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
22. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
23. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
25. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
26. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
27. ? @ 0x00007f3c91188ac3
28. ? @ 0x00007f3c9121a850

2024.11.05 19:49:09.958118 [ 2931 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:11.268045 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:12.818000 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:14.350657 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/ugg/fgqnusxkuxgpnisfnadwkxmttenef, total size 218103808, count 218103808, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 809eea07-98e6-434c-8a84-a03e816a2583, upload has finished false.
2024.11.05 19:49:14.351501 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/ugg/fgqnusxkuxgpnisfnadwkxmttenef, total size 218103808, count 218103808, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 809eea07-98e6-434c-8a84-a03e816a2583, upload has finished false
2024.11.05 19:49:14.424934 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/ugg/fgqnusxkuxgpnisfnadwkxmttenef, total size 218103808, count 218103808, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 809eea07-98e6-434c-8a84-a03e816a2583, upload has finished false
2024.11.05 19:49:14.480409 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 48 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:14.624506 [ 5215 ] {9ce1c83c-2428-4072-90f3-a8fc1a9dfe01} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=938, jobs=884). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:53826) (comment: 01451_normalize_query.sql) (in query: SELECT normalizeQuery('SELECT 1 xyz11');), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001d5394f3
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:112: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000001d539234
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000001d5395e9
12. /build/src/Server/TCPHandler.cpp:1023: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b57f
13. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:49:14.649421 [ 5215 ] {9ce1c83c-2428-4072-90f3-a8fc1a9dfe01} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=938, jobs=884). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001d5394f3
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:112: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000001d539234
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000001d5395e9
12. /build/src/Server/TCPHandler.cpp:1023: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b57f
13. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:49:15.147808 [ 2937 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:18.598468 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0a233324-63e4-4351-8154-d23ef4927e18::202411_1_66_13}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
24. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
25. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
26. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
27. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:18.640467 [ 5795 ] {75eee3d8-5cde-42df-a90e-2e198f8633be} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:42156) (comment: 00154_avro.sql) (in query: SELECT sum(cityHash64(*)) FROM (SELECT * FROM test.hits LIMIT 10000);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:49:18.799857 [ 5795 ] {75eee3d8-5cde-42df-a90e-2e198f8633be} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:49:19.291214 [ 2939 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:19.305569 [ 5215 ] {7959854e-1468-4d81-aed3-6b8b923d06f6} <Error> executeQuery: Code: 80. DB::Exception: It's not allowed to specify explicit zookeeper_path and replica_name for ReplicatedMergeTree arguments in Replicated database. If you really want to specify them explicitly, enable setting database_replicated_allow_replicated_engine_arguments. (INCORRECT_QUERY) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:36744) (comment: 02514_database_replicated_no_arguments_for_rmt.sh) (in query: CREATE TABLE test_15_db.tab_rmt_fail (`x` UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/02514_database_replicated_no_arguments_for_rmt_test_15/root/{shard}', '{replica}') ORDER BY x SETTINGS index_granularity = 32767, min_bytes_for_wide_part = 0, ratio_of_defaults_for_sparse_serialization = 0.25623589754104614, replace_long_file_name_to_hash = true, max_file_name_length = 128, min_bytes_for_full_part_storage = 387784452, compact_parts_max_bytes_to_buffer = 470784608, compact_parts_max_granules_to_buffer = 87, compact_parts_merge_max_bytes_to_prefetch_part = 16400833, merge_max_block_size = 19443, old_parts_lifetime = 412., prefer_fetch_merged_part_size_threshold = 10737418240, vertical_merge_algorithm_min_rows_to_activate = 688631, vertical_merge_algorithm_min_columns_to_activate = 1, min_merge_bytes_to_use_direct_io = 7816243294, index_granularity_bytes = 24526118, concurrent_part_removal_threshold = 2, allow_vertical_merges_from_compact_to_wide_parts = false, cache_populated_by_fetch = true, marks_compress_block_size = 28573, primary_key_compress_block_size = 44391), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Databases/DatabaseReplicated.cpp:717: DB::DatabaseReplicated::checkQueryValid(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>) const @ 0x000000001a0f5a65
7. /build/src/Databases/DatabaseReplicated.cpp:776: DB::DatabaseReplicated::tryEnqueueReplicatedDDL(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, DB::QueryFlags) @ 0x000000001a0f65b7
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:1294: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b018701
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
10. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
11. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
12. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
13. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
14. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
15. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
16. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
17. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
18. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
19. ? @ 0x00007f3c91188ac3
20. ? @ 0x00007f3c9121a850

2024.11.05 19:49:19.310232 [ 5215 ] {7959854e-1468-4d81-aed3-6b8b923d06f6} <Error> TCPHandler: Code: 80. DB::Exception: It's not allowed to specify explicit zookeeper_path and replica_name for ReplicatedMergeTree arguments in Replicated database. If you really want to specify them explicitly, enable setting database_replicated_allow_replicated_engine_arguments. (INCORRECT_QUERY), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Databases/DatabaseReplicated.cpp:717: DB::DatabaseReplicated::checkQueryValid(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>) const @ 0x000000001a0f5a65
7. /build/src/Databases/DatabaseReplicated.cpp:776: DB::DatabaseReplicated::tryEnqueueReplicatedDDL(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, DB::QueryFlags) @ 0x000000001a0f65b7
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:1294: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b018701
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
10. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
11. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
12. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
13. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
14. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
15. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
16. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
17. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
18. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
19. ? @ 0x00007f3c91188ac3
20. ? @ 0x00007f3c9121a850

2024.11.05 19:49:19.327068 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0a233324-63e4-4351-8154-d23ef4927e18::202411_1_66_13}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:23.240739 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {62ec4990-945b-4d8e-9889-ca33c67862f8::202411_1_76_15}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:24.256095 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {62ec4990-945b-4d8e-9889-ca33c67862f8::202411_1_76_15}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:26.628360 [ 2931 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:27.024592 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/toy/jzescugduhvukvafqatpdbtcwxdns, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id fa252cfd-8103-4f13-8cc3-b4cdae7b5571, upload has finished false.
2024.11.05 19:49:27.025269 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/toy/jzescugduhvukvafqatpdbtcwxdns, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id fa252cfd-8103-4f13-8cc3-b4cdae7b5571, upload has finished false
2024.11.05 19:49:27.086949 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/toy/jzescugduhvukvafqatpdbtcwxdns, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id fa252cfd-8103-4f13-8cc3-b4cdae7b5571, upload has finished false
2024.11.05 19:49:27.157088 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 32 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:27.197009 [ 5798 ] {38ff869a-5b5b-477a-b4ab-52ff2c203f86} <Error> executeQuery: Code: 164. DB::Exception: default: Cannot execute query in readonly mode. For queries over HTTP, method GET implies readonly. You should use method POST for modifying queries. (READONLY) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55716) (comment: 00305_http_and_readonly.sh) (in query: CREATE TABLE table_00305a(x Int8) ENGINE=Log ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Access/ContextAccess.cpp:546: auto bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, String>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, String const&) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x000000001958db0a
7. /build/src/Access/ContextAccess.cpp:621: bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, String>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, String const&) const @ 0x000000001958b387
8. /build/src/Access/ContextAccess.cpp:657: bool DB::ContextAccess::checkAccessImpl<true, false, String>(DB::AccessFlags const&, std::basic_string_view<char, std::char_traits<char>>, String const&) const @ 0x000000001958ab4c
9. /build/src/Access/ContextAccess.cpp:676: bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessRightsElement const&) const @ 0x0000000019584755
10. /build/src/Access/ContextAccess.cpp:693: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessRightsElement const&) const @ 0x000000001953fe52
11. /build/src/Access/ContextAccess.cpp:701: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessRightsElements const&) const @ 0x000000001953feee
12. /build/src/Access/ContextAccess.cpp:731: DB::ContextAccess::checkAccess(DB::AccessRightsElements const&) const @ 0x000000001953b05d
13. /build/src/Interpreters/Context.cpp:1398: void DB::Context::checkAccessImpl<DB::AccessRightsElements>(DB::AccessRightsElements const&) const @ 0x000000001a56ff16
14. /build/src/Interpreters/Context.cpp:1412: DB::Context::checkAccess(DB::AccessRightsElements const&) const @ 0x000000001a5454dd
15. /build/src/Interpreters/InterpreterCreateQuery.cpp:1785: DB::InterpreterCreateQuery::execute() @ 0x000000001b0217f8
16. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
17. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
18. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
19. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
20. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
21. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
22. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
23. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
24. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
25. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
26. ? @ 0x00007f3c91188ac3
27. ? @ 0x00007f3c9121a850

2024.11.05 19:49:27.227710 [ 5798 ] {38ff869a-5b5b-477a-b4ab-52ff2c203f86} <Error> DynamicQueryHandler: Code: 164. DB::Exception: default: Cannot execute query in readonly mode. For queries over HTTP, method GET implies readonly. You should use method POST for modifying queries. (READONLY), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Access/ContextAccess.cpp:546: auto bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, String>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, String const&) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x000000001958db0a
7. /build/src/Access/ContextAccess.cpp:621: bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, String>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, String const&) const @ 0x000000001958b387
8. /build/src/Access/ContextAccess.cpp:657: bool DB::ContextAccess::checkAccessImpl<true, false, String>(DB::AccessFlags const&, std::basic_string_view<char, std::char_traits<char>>, String const&) const @ 0x000000001958ab4c
9. /build/src/Access/ContextAccess.cpp:676: bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessRightsElement const&) const @ 0x0000000019584755
10. /build/src/Access/ContextAccess.cpp:693: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessRightsElement const&) const @ 0x000000001953fe52
11. /build/src/Access/ContextAccess.cpp:701: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessRightsElements const&) const @ 0x000000001953feee
12. /build/src/Access/ContextAccess.cpp:731: DB::ContextAccess::checkAccess(DB::AccessRightsElements const&) const @ 0x000000001953b05d
13. /build/src/Interpreters/Context.cpp:1398: void DB::Context::checkAccessImpl<DB::AccessRightsElements>(DB::AccessRightsElements const&) const @ 0x000000001a56ff16
14. /build/src/Interpreters/Context.cpp:1412: DB::Context::checkAccess(DB::AccessRightsElements const&) const @ 0x000000001a5454dd
15. /build/src/Interpreters/InterpreterCreateQuery.cpp:1785: DB::InterpreterCreateQuery::execute() @ 0x000000001b0217f8
16. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
17. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
18. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
19. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
20. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
21. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
22. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
23. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
24. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
25. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
26. ? @ 0x00007f3c91188ac3
27. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:27.704613 [ 5215 ] {} <Error> DynamicQueryHandler: Code: 164. DB::Exception: Cannot modify 'readonly' setting in readonly mode. (READONLY), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:89: DB::Exception::Exception(String const&, int, bool) @ 0x000000000a9434aa
5. /build/src/Common/Exception.h:58: DB::Exception::Exception(PreformattedMessage const&, int) @ 0x0000000018d1d809
6. /build/src/Access/SettingsConstraints.cpp:307: DB::SettingsConstraints::Checker::check(DB::SettingChange&, DB::Field const&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x00000000197482ba
7. /build/src/Access/SettingsConstraints.cpp:288: DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000019747c0d
8. /build/src/Access/SettingsConstraints.cpp:197: DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const::$_0::operator()(DB::SettingChange&) const @ 0x000000001974b62d
9. /build/contrib/llvm-project/libcxx/include/__algorithm/remove_if.h:32: std::__wrap_iter<DB::SettingChange*> std::remove_if<std::__wrap_iter<DB::SettingChange*>, DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const::$_0>(std::__wrap_iter<DB::SettingChange*>, std::__wrap_iter<DB::SettingChange*>, DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const::$_0) @ 0x000000001974b502
10. /build/contrib/boost/boost/range/algorithm_ext/erase.hpp:50: DB::SettingsChanges& boost::range::remove_erase_if<DB::SettingsChanges, DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const::$_0>(DB::SettingsChanges&, DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const::$_0) @ 0x0000000019747e41
11. /build/src/Access/SettingsConstraints.cpp:199: DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000019747db5
12. /build/src/Interpreters/Context.cpp:2197: DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x000000001a54ca34
13. /build/src/Interpreters/Context.cpp:2236: DB::Context::checkSettingsConstraints(DB::SettingsChanges&, DB::SettingSource) @ 0x000000001a54d0db
14. /build/src/Server/HTTPHandler.cpp:796: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3da1ea
15. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
16. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:28.869170 [ 5793 ] {3ca2662b-c885-4e74-89d0-a2d8edd94981} <Error> executeQuery: Code: 164. DB::Exception: default: Cannot execute query in readonly mode. (READONLY) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55730) (comment: 00305_http_and_readonly.sh) (in query: CREATE TABLE table_00305c(x Int8) ENGINE=Log ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Access/ContextAccess.cpp:546: auto bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, String>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, String const&) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x000000001958db0a
7. /build/src/Access/ContextAccess.cpp:627: bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, String>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, String const&) const @ 0x000000001958b421
8. /build/src/Access/ContextAccess.cpp:657: bool DB::ContextAccess::checkAccessImpl<true, false, String>(DB::AccessFlags const&, std::basic_string_view<char, std::char_traits<char>>, String const&) const @ 0x000000001958ab4c
9. /build/src/Access/ContextAccess.cpp:676: bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessRightsElement const&) const @ 0x0000000019584755
10. /build/src/Access/ContextAccess.cpp:693: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessRightsElement const&) const @ 0x000000001953fe52
11. /build/src/Access/ContextAccess.cpp:701: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessRightsElements const&) const @ 0x000000001953feee
12. /build/src/Access/ContextAccess.cpp:731: DB::ContextAccess::checkAccess(DB::AccessRightsElements const&) const @ 0x000000001953b05d
13. /build/src/Interpreters/Context.cpp:1398: void DB::Context::checkAccessImpl<DB::AccessRightsElements>(DB::AccessRightsElements const&) const @ 0x000000001a56ff16
14. /build/src/Interpreters/Context.cpp:1412: DB::Context::checkAccess(DB::AccessRightsElements const&) const @ 0x000000001a5454dd
15. /build/src/Interpreters/InterpreterCreateQuery.cpp:1785: DB::InterpreterCreateQuery::execute() @ 0x000000001b0217f8
16. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
17. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
18. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
19. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
20. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
21. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
22. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
23. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
24. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
25. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
26. ? @ 0x00007f3c91188ac3
27. ? @ 0x00007f3c9121a850

2024.11.05 19:49:28.886777 [ 5793 ] {3ca2662b-c885-4e74-89d0-a2d8edd94981} <Error> DynamicQueryHandler: Code: 164. DB::Exception: default: Cannot execute query in readonly mode. (READONLY), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Access/ContextAccess.cpp:546: auto bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, String>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, String const&) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x000000001958db0a
7. /build/src/Access/ContextAccess.cpp:627: bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, String>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, String const&) const @ 0x000000001958b421
8. /build/src/Access/ContextAccess.cpp:657: bool DB::ContextAccess::checkAccessImpl<true, false, String>(DB::AccessFlags const&, std::basic_string_view<char, std::char_traits<char>>, String const&) const @ 0x000000001958ab4c
9. /build/src/Access/ContextAccess.cpp:676: bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessRightsElement const&) const @ 0x0000000019584755
10. /build/src/Access/ContextAccess.cpp:693: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessRightsElement const&) const @ 0x000000001953fe52
11. /build/src/Access/ContextAccess.cpp:701: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessRightsElements const&) const @ 0x000000001953feee
12. /build/src/Access/ContextAccess.cpp:731: DB::ContextAccess::checkAccess(DB::AccessRightsElements const&) const @ 0x000000001953b05d
13. /build/src/Interpreters/Context.cpp:1398: void DB::Context::checkAccessImpl<DB::AccessRightsElements>(DB::AccessRightsElements const&) const @ 0x000000001a56ff16
14. /build/src/Interpreters/Context.cpp:1412: DB::Context::checkAccess(DB::AccessRightsElements const&) const @ 0x000000001a5454dd
15. /build/src/Interpreters/InterpreterCreateQuery.cpp:1785: DB::InterpreterCreateQuery::execute() @ 0x000000001b0217f8
16. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
17. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
18. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
19. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
20. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
21. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
22. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
23. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
24. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
25. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
26. ? @ 0x00007f3c91188ac3
27. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:29.177233 [ 5215 ] {8f611eeb-ac3c-4faa-b301-bae859316931} <Error> executeQuery: Code: 164. DB::Exception: default: Cannot execute query in readonly mode. (READONLY) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55746) (comment: 00305_http_and_readonly.sh) (in query: CREATE TABLE table_00305c(x Int8) ENGINE=Log ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Access/ContextAccess.cpp:546: auto bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, String>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, String const&) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x000000001958db0a
7. /build/src/Access/ContextAccess.cpp:627: bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, String>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, String const&) const @ 0x000000001958b421
8. /build/src/Access/ContextAccess.cpp:657: bool DB::ContextAccess::checkAccessImpl<true, false, String>(DB::AccessFlags const&, std::basic_string_view<char, std::char_traits<char>>, String const&) const @ 0x000000001958ab4c
9. /build/src/Access/ContextAccess.cpp:676: bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessRightsElement const&) const @ 0x0000000019584755
10. /build/src/Access/ContextAccess.cpp:693: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessRightsElement const&) const @ 0x000000001953fe52
11. /build/src/Access/ContextAccess.cpp:701: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessRightsElements const&) const @ 0x000000001953feee
12. /build/src/Access/ContextAccess.cpp:731: DB::ContextAccess::checkAccess(DB::AccessRightsElements const&) const @ 0x000000001953b05d
13. /build/src/Interpreters/Context.cpp:1398: void DB::Context::checkAccessImpl<DB::AccessRightsElements>(DB::AccessRightsElements const&) const @ 0x000000001a56ff16
14. /build/src/Interpreters/Context.cpp:1412: DB::Context::checkAccess(DB::AccessRightsElements const&) const @ 0x000000001a5454dd
15. /build/src/Interpreters/InterpreterCreateQuery.cpp:1785: DB::InterpreterCreateQuery::execute() @ 0x000000001b0217f8
16. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
17. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
18. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
19. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
20. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
21. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
22. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
23. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
24. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
25. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
26. ? @ 0x00007f3c91188ac3
27. ? @ 0x00007f3c9121a850

2024.11.05 19:49:29.183593 [ 5215 ] {8f611eeb-ac3c-4faa-b301-bae859316931} <Error> DynamicQueryHandler: Code: 164. DB::Exception: default: Cannot execute query in readonly mode. (READONLY), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Access/ContextAccess.cpp:546: auto bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, String>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, String const&) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x000000001958db0a
7. /build/src/Access/ContextAccess.cpp:627: bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, String>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, String const&) const @ 0x000000001958b421
8. /build/src/Access/ContextAccess.cpp:657: bool DB::ContextAccess::checkAccessImpl<true, false, String>(DB::AccessFlags const&, std::basic_string_view<char, std::char_traits<char>>, String const&) const @ 0x000000001958ab4c
9. /build/src/Access/ContextAccess.cpp:676: bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessRightsElement const&) const @ 0x0000000019584755
10. /build/src/Access/ContextAccess.cpp:693: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessRightsElement const&) const @ 0x000000001953fe52
11. /build/src/Access/ContextAccess.cpp:701: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessRightsElements const&) const @ 0x000000001953feee
12. /build/src/Access/ContextAccess.cpp:731: DB::ContextAccess::checkAccess(DB::AccessRightsElements const&) const @ 0x000000001953b05d
13. /build/src/Interpreters/Context.cpp:1398: void DB::Context::checkAccessImpl<DB::AccessRightsElements>(DB::AccessRightsElements const&) const @ 0x000000001a56ff16
14. /build/src/Interpreters/Context.cpp:1412: DB::Context::checkAccess(DB::AccessRightsElements const&) const @ 0x000000001a5454dd
15. /build/src/Interpreters/InterpreterCreateQuery.cpp:1785: DB::InterpreterCreateQuery::execute() @ 0x000000001b0217f8
16. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
17. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
18. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
19. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
20. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
21. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
22. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
23. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
24. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
25. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
26. ? @ 0x00007f3c91188ac3
27. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:29.605271 [ 5793 ] {fbe39da0-3922-45a6-88c1-8a4fcfb4e54b} <Error> executeQuery: Code: 164. DB::Exception: default: Cannot execute query in readonly mode. For queries over HTTP, method GET implies readonly. You should use method POST for modifying queries. (READONLY) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55762) (comment: 00305_http_and_readonly.sh) (in query: DROP TABLE table_00305a ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Access/ContextAccess.cpp:546: auto bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, std::basic_string_view<char, std::char_traits<char>> const&) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x000000001957b429
7. /build/src/Access/ContextAccess.cpp:621: bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, std::basic_string_view<char, std::char_traits<char>> const&) const @ 0x0000000019578cc7
8. /build/src/Access/ContextAccess.cpp:657: bool DB::ContextAccess::checkAccessImpl<true, false, std::basic_string_view<char, std::char_traits<char>>>(DB::AccessFlags const&, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>> const&) const @ 0x000000001953fb4c
9. /build/src/Access/ContextAccess.cpp:726: DB::ContextAccess::checkAccess(DB::AccessFlags const&, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>) const @ 0x000000001953aec9
10. /build/src/Interpreters/Context.cpp:1398: void DB::Context::checkAccessImpl<DB::AccessFlags, String, String>(DB::AccessFlags const&, String const&, String const&) const @ 0x000000001a56fb7d
11. /build/src/Interpreters/Context.cpp:1407: DB::Context::checkAccess(DB::AccessFlags const&, DB::StorageID const&) const @ 0x000000001a5451da
12. /build/src/Interpreters/InterpreterDropQuery.cpp:256: DB::InterpreterDropQuery::executeToTableImpl(std::shared_ptr<DB::Context const>, DB::ASTDropQuery&, std::shared_ptr<DB::IDatabase>&, StrongTypedef<wide::integer<128ul, unsigned int>, DB::UUIDTag>&) @ 0x000000001b03df92
13. /build/src/Interpreters/InterpreterDropQuery.cpp:102: DB::InterpreterDropQuery::executeToTable(DB::ASTDropQuery&) @ 0x000000001b03bd94
14. /build/src/Interpreters/InterpreterDropQuery.cpp:73: DB::InterpreterDropQuery::execute() @ 0x000000001b03b34e
15. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
16. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
17. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
18. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
19. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
20. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
21. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
22. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
23. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
24. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
25. ? @ 0x00007f3c91188ac3
26. ? @ 0x00007f3c9121a850

2024.11.05 19:49:29.625987 [ 5793 ] {fbe39da0-3922-45a6-88c1-8a4fcfb4e54b} <Error> DynamicQueryHandler: Code: 164. DB::Exception: default: Cannot execute query in readonly mode. For queries over HTTP, method GET implies readonly. You should use method POST for modifying queries. (READONLY), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Access/ContextAccess.cpp:546: auto bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, std::basic_string_view<char, std::char_traits<char>> const&) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x000000001957b429
7. /build/src/Access/ContextAccess.cpp:621: bool DB::ContextAccess::checkAccessImplHelper<true, false, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>>(DB::AccessFlags, std::basic_string_view<char, std::char_traits<char>> const&, std::basic_string_view<char, std::char_traits<char>> const&) const @ 0x0000000019578cc7
8. /build/src/Access/ContextAccess.cpp:657: bool DB::ContextAccess::checkAccessImpl<true, false, std::basic_string_view<char, std::char_traits<char>>>(DB::AccessFlags const&, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>> const&) const @ 0x000000001953fb4c
9. /build/src/Access/ContextAccess.cpp:726: DB::ContextAccess::checkAccess(DB::AccessFlags const&, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>) const @ 0x000000001953aec9
10. /build/src/Interpreters/Context.cpp:1398: void DB::Context::checkAccessImpl<DB::AccessFlags, String, String>(DB::AccessFlags const&, String const&, String const&) const @ 0x000000001a56fb7d
11. /build/src/Interpreters/Context.cpp:1407: DB::Context::checkAccess(DB::AccessFlags const&, DB::StorageID const&) const @ 0x000000001a5451da
12. /build/src/Interpreters/InterpreterDropQuery.cpp:256: DB::InterpreterDropQuery::executeToTableImpl(std::shared_ptr<DB::Context const>, DB::ASTDropQuery&, std::shared_ptr<DB::IDatabase>&, StrongTypedef<wide::integer<128ul, unsigned int>, DB::UUIDTag>&) @ 0x000000001b03df92
13. /build/src/Interpreters/InterpreterDropQuery.cpp:102: DB::InterpreterDropQuery::executeToTable(DB::ASTDropQuery&) @ 0x000000001b03bd94
14. /build/src/Interpreters/InterpreterDropQuery.cpp:73: DB::InterpreterDropQuery::execute() @ 0x000000001b03b34e
15. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
16. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
17. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
18. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
19. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
20. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
21. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
22. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
23. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
24. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
25. ? @ 0x00007f3c91188ac3
26. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:29.712663 [ 2934 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:30.726629 [ 2932 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:31.065041 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 6 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:31.112113 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:34.330341 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 2 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:35.388453 [ 5798 ] {c4a762a5-a974-452c-9bd8-f94fb7e1ed7d} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46924) (comment: 00910_zookeeper_test_alter_compression_codecs_long.sql) (in query: CREATE TABLE alter_compression_codec1 (`somedate` Date CODEC(LZ4), `id` UInt64 CODEC(NONE)) ENGINE = ReplicatedMergeTree(concat('/clickhouse/tables/{database}/test_00910/', currentDatabase(), 'alter_compression_codecs/{shard}'), '1_{replica}') PARTITION BY somedate ORDER BY id SETTINGS index_granularity = 21324, min_bytes_for_wide_part = 1073741824, ratio_of_defaults_for_sparse_serialization = 0.9405815005302429, replace_long_file_name_to_hash = true, max_file_name_length = 0, min_bytes_for_full_part_storage = 536870912, compact_parts_max_bytes_to_buffer = 76713441, compact_parts_max_granules_to_buffer = 18, compact_parts_merge_max_bytes_to_prefetch_part = 25362746, merge_max_block_size = 2349, old_parts_lifetime = 480., prefer_fetch_merged_part_size_threshold = 10737418240, vertical_merge_algorithm_min_rows_to_activate = 1, vertical_merge_algorithm_min_columns_to_activate = 100, min_merge_bytes_to_use_direct_io = 3978188406, index_granularity_bytes = 20224473, concurrent_part_removal_threshold = 100, allow_vertical_merges_from_compact_to_wide_parts = false, cache_populated_by_fetch = true, marks_compress_block_size = 81072, primary_key_compress_block_size = 15391), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/MergeTree/MergeTreeData.cpp:1703: DB::MergeTreeData::loadDataParts(bool, std::optional<std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>>) @ 0x000000001cbc7077
11. /build/src/Storages/StorageReplicatedMergeTree.cpp:445: DB::StorageReplicatedMergeTree::StorageReplicatedMergeTree(String const&, String const&, DB::LoadingStrictnessLevel, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata const&, std::shared_ptr<DB::Context>, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::RenamingRestrictions, bool) @ 0x000000001c4b1748
12. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageReplicatedMergeTree* std::construct_at[abi:v15000]<DB::StorageReplicatedMergeTree, String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::RenamingRestrictions&, bool&, DB::StorageReplicatedMergeTree*>(DB::StorageReplicatedMergeTree*, String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&, DB::RenamingRestrictions&, bool&) @ 0x000000001d0ea838
13. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageReplicatedMergeTree>>::construct[abi:v15000]<DB::StorageReplicatedMergeTree, String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::RenamingRestrictions&, bool&, void, void>(std::allocator<DB::StorageReplicatedMergeTree>&, DB::StorageReplicatedMergeTree*, String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&, DB::RenamingRestrictions&, bool&) @ 0x000000001d0ea502
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageReplicatedMergeTree, std::allocator<DB::StorageReplicatedMergeTree>>::__shared_ptr_emplace[abi:v15000]<String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::RenamingRestrictions&, bool&>(std::allocator<DB::StorageReplicatedMergeTree>, String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&, DB::RenamingRestrictions&, bool&) @ 0x000000001d0ea154
15. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageReplicatedMergeTree> std::allocate_shared[abi:v15000]<DB::StorageReplicatedMergeTree, std::allocator<DB::StorageReplicatedMergeTree>, String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::RenamingRestrictions&, bool&, void>(std::allocator<DB::StorageReplicatedMergeTree> const&, String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&, DB::RenamingRestrictions&, bool&) @ 0x000000001d0e9f1f
16. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageReplicatedMergeTree> std::make_shared[abi:v15000]<DB::StorageReplicatedMergeTree, String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::RenamingRestrictions&, bool&, void>(String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&, DB::RenamingRestrictions&, bool&) @ 0x000000001d0e7c88
17. /build/src/Storages/MergeTree/registerStorageMergeTree.cpp:726: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000001d0e5808
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000193086fd
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000193086bd
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001930867d
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019308645
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
24. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
25. /build/src/Interpreters/InterpreterCreateQuery.cpp:1493: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x000000001b01ec0c
26. /build/src/Interpreters/InterpreterCreateQuery.cpp:1313: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b018b66
27. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
28. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
29. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
30. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
31. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5

2024.11.05 19:49:35.391311 [ 2940 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:35.419516 [ 5798 ] {c4a762a5-a974-452c-9bd8-f94fb7e1ed7d} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/MergeTree/MergeTreeData.cpp:1703: DB::MergeTreeData::loadDataParts(bool, std::optional<std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>>) @ 0x000000001cbc7077
11. /build/src/Storages/StorageReplicatedMergeTree.cpp:445: DB::StorageReplicatedMergeTree::StorageReplicatedMergeTree(String const&, String const&, DB::LoadingStrictnessLevel, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata const&, std::shared_ptr<DB::Context>, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::RenamingRestrictions, bool) @ 0x000000001c4b1748
12. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageReplicatedMergeTree* std::construct_at[abi:v15000]<DB::StorageReplicatedMergeTree, String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::RenamingRestrictions&, bool&, DB::StorageReplicatedMergeTree*>(DB::StorageReplicatedMergeTree*, String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&, DB::RenamingRestrictions&, bool&) @ 0x000000001d0ea838
13. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageReplicatedMergeTree>>::construct[abi:v15000]<DB::StorageReplicatedMergeTree, String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::RenamingRestrictions&, bool&, void, void>(std::allocator<DB::StorageReplicatedMergeTree>&, DB::StorageReplicatedMergeTree*, String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&, DB::RenamingRestrictions&, bool&) @ 0x000000001d0ea502
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageReplicatedMergeTree, std::allocator<DB::StorageReplicatedMergeTree>>::__shared_ptr_emplace[abi:v15000]<String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::RenamingRestrictions&, bool&>(std::allocator<DB::StorageReplicatedMergeTree>, String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&, DB::RenamingRestrictions&, bool&) @ 0x000000001d0ea154
15. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageReplicatedMergeTree> std::allocate_shared[abi:v15000]<DB::StorageReplicatedMergeTree, std::allocator<DB::StorageReplicatedMergeTree>, String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::RenamingRestrictions&, bool&, void>(std::allocator<DB::StorageReplicatedMergeTree> const&, String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&, DB::RenamingRestrictions&, bool&) @ 0x000000001d0e9f1f
16. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageReplicatedMergeTree> std::make_shared[abi:v15000]<DB::StorageReplicatedMergeTree, String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::RenamingRestrictions&, bool&, void>(String&, String&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&, DB::RenamingRestrictions&, bool&) @ 0x000000001d0e7c88
17. /build/src/Storages/MergeTree/registerStorageMergeTree.cpp:726: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000001d0e5808
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000193086fd
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000193086bd
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001930867d
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019308645
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
24. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
25. /build/src/Interpreters/InterpreterCreateQuery.cpp:1493: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x000000001b01ec0c
26. /build/src/Interpreters/InterpreterCreateQuery.cpp:1313: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b018b66
27. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
28. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
29. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
30. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
31. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5

2024.11.05 19:49:35.444519 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {29fdad69-1244-4c25-af3c-70c43e910f15::202411_1_119_22}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=9). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:38.558165 [ 2926 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:39.325312 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {d830f0a7-c4af-4441-961e-a7e3a729e583::202411_1_81_16}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
25. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
26. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
27. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
28. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:39.600262 [ 2927 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:44.204219 [ 3584 ] {1be9bc8e-3564-4fed-a9c2-0f344e0f8858::1_3_3_0_5} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:44.391442 [ 5215 ] {63fd05d0-173d-4f61-8561-130f567e71ae} <Error> executeQuery: Code: 341. DB::Exception: Exception happened during execution of mutation 'mutation_5.txt' with part '1_3_3_0' reason: 'Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build))'. This error maybe retryable or not. In case of unretryable error, mutation can be killed with KILL MUTATION query. (UNFINISHED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46936) (comment: 02399_merge_tree_mutate_in_partition.sql) (in query: alter table mt update n = n + (n not in m) in partition id '1' where 1 settings mutations_sync=1;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*, String, String&, String&>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, char const*&&, String&&, String&, String&) @ 0x000000001ce28604
6. /build/src/Storages/MergeTree/MergeTreeMutationStatus.cpp:26: DB::checkMutationStatus(std::optional<DB::MergeTreeMutationStatus>&, std::set<String, std::less<String>, std::allocator<String>> const&) @ 0x000000001ce283a0
7. /build/src/Storages/StorageMergeTree.cpp:620: DB::StorageMergeTree::waitForMutation(long, String const&, bool) @ 0x000000001d0f3a12
8. /build/src/Storages/StorageMergeTree.cpp:590: DB::StorageMergeTree::waitForMutation(long, bool) @ 0x000000001d0f1068
9. /build/src/Storages/StorageMergeTree.cpp:661: DB::StorageMergeTree::mutate(DB::MutationCommands const&, std::shared_ptr<DB::Context const>) @ 0x000000001d0f5269
10. /build/src/Interpreters/InterpreterAlterQuery.cpp:221: DB::InterpreterAlterQuery::executeToTable(DB::ASTAlterQuery const&) @ 0x000000001affb000
11. /build/src/Interpreters/InterpreterAlterQuery.cpp:67: DB::InterpreterAlterQuery::execute() @ 0x000000001aff84de
12. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
13. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
14. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
15. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:49:44.391991 [ 3584 ] {1be9bc8e-3564-4fed-a9c2-0f344e0f8858::1_3_3_0_5} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:44.411050 [ 5215 ] {63fd05d0-173d-4f61-8561-130f567e71ae} <Error> TCPHandler: Code: 341. DB::Exception: Exception happened during execution of mutation 'mutation_5.txt' with part '1_3_3_0' reason: 'Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build))'. This error maybe retryable or not. In case of unretryable error, mutation can be killed with KILL MUTATION query. (UNFINISHED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*, String, String&, String&>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, char const*&&, String&&, String&, String&) @ 0x000000001ce28604
6. /build/src/Storages/MergeTree/MergeTreeMutationStatus.cpp:26: DB::checkMutationStatus(std::optional<DB::MergeTreeMutationStatus>&, std::set<String, std::less<String>, std::allocator<String>> const&) @ 0x000000001ce283a0
7. /build/src/Storages/StorageMergeTree.cpp:620: DB::StorageMergeTree::waitForMutation(long, String const&, bool) @ 0x000000001d0f3a12
8. /build/src/Storages/StorageMergeTree.cpp:590: DB::StorageMergeTree::waitForMutation(long, bool) @ 0x000000001d0f1068
9. /build/src/Storages/StorageMergeTree.cpp:661: DB::StorageMergeTree::mutate(DB::MutationCommands const&, std::shared_ptr<DB::Context const>) @ 0x000000001d0f5269
10. /build/src/Interpreters/InterpreterAlterQuery.cpp:221: DB::InterpreterAlterQuery::executeToTable(DB::ASTAlterQuery const&) @ 0x000000001affb000
11. /build/src/Interpreters/InterpreterAlterQuery.cpp:67: DB::InterpreterAlterQuery::execute() @ 0x000000001aff84de
12. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
13. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
14. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
15. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:49:45.102911 [ 3571 ] {1be9bc8e-3564-4fed-a9c2-0f344e0f8858::1_1_1_0_5} <Error> MutatePlainMergeTreeTask: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:45.454640 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:45.456526 [ 3571 ] {1be9bc8e-3564-4fed-a9c2-0f344e0f8858::1_1_1_0_5} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MutateTask.cpp:1685: DB::MutateAllPartColumnsTask::finalize() @ 0x000000001cf750bd
24. /build/src/Storages/MergeTree/MutateTask.cpp:1419: DB::MutateAllPartColumnsTask::executeStep() @ 0x000000001cf71543
25. /build/src/Storages/MergeTree/MutateTask.cpp:2010: DB::MutateTask::execute() @ 0x000000001cf4e38f
26. /build/src/Storages/MergeTree/MutatePlainMergeTreeTask.cpp:90: DB::MutatePlainMergeTreeTask::executeStep() @ 0x000000001d141e01
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:45.699672 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:48.989427 [ 2937 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:49.622408 [ 5798 ] {83f6b0c4-d6b8-4844-9aff-29be1ca5e754} <Error> executeQuery: Code: 497. DB::Exception: user_test_02337: Not enough privileges. To execute this query, it's necessary to have the grant SYSTEM DROP FILESYSTEM CACHE ON *.*. (ACCESS_DENIED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:56136) (comment: 02337_drop_filesystem_cache_access.sh) (in query: SYSTEM DROP FILESYSTEM CACHE;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000a8f2fcc
6. /build/src/Access/ContextAccess.cpp:546: auto bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessFlags) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<String>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x0000000019574851
7. /build/src/Access/ContextAccess.cpp:586: bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessFlags) const @ 0x0000000019572e73
8. /build/src/Access/ContextAccess.cpp:651: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessFlags const&) const @ 0x000000001953f982
9. /build/src/Access/ContextAccess.cpp:724: DB::ContextAccess::checkAccess(DB::AccessFlags const&) const @ 0x000000001953adfd
10. /build/src/Interpreters/Context.cpp:1398: void DB::Context::checkAccessImpl<DB::AccessFlags>(DB::AccessFlags const&) const @ 0x000000001a56f756
11. /build/src/Interpreters/Context.cpp:1401: DB::Context::checkAccess(DB::AccessFlags const&) const @ 0x000000001a544fbd
12. /build/src/Interpreters/InterpreterSystemQuery.cpp:393: DB::InterpreterSystemQuery::execute() @ 0x000000001bbee6c8
13. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
14. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
15. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
16. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850

2024.11.05 19:49:49.638797 [ 5215 ] {619b8190-0e2b-4128-b274-298d068be0ea} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=938, jobs=882). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:56134) (comment: 01780_column_sparse_tuple.sql) (in query: INSERT INTO sparse_tuple SELECT number, (if (number % 20 = 0, number, 0), repeat('a', number % 10 + 1)) FROM numbers(1000);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&) @ 0x000000001d512c01
10. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:92: DB::CompletedPipelineExecutor::execute() @ 0x000000001d512816
11. /build/src/Server/TCPHandler.cpp:566: DB::TCPHandler::runImpl() @ 0x000000001d4709b1
12. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
13. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
14. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
15. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
16. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
17. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
18. ? @ 0x00007f3c91188ac3
19. ? @ 0x00007f3c9121a850

2024.11.05 19:49:49.679170 [ 5215 ] {619b8190-0e2b-4128-b274-298d068be0ea} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=938, jobs=882). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&) @ 0x000000001d512c01
10. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:92: DB::CompletedPipelineExecutor::execute() @ 0x000000001d512816
11. /build/src/Server/TCPHandler.cpp:566: DB::TCPHandler::runImpl() @ 0x000000001d4709b1
12. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
13. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
14. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
15. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
16. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
17. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
18. ? @ 0x00007f3c91188ac3
19. ? @ 0x00007f3c9121a850

2024.11.05 19:49:49.711568 [ 5798 ] {83f6b0c4-d6b8-4844-9aff-29be1ca5e754} <Error> TCPHandler: Code: 497. DB::Exception: user_test_02337: Not enough privileges. To execute this query, it's necessary to have the grant SYSTEM DROP FILESYSTEM CACHE ON *.*. (ACCESS_DENIED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000a8f2fcc
6. /build/src/Access/ContextAccess.cpp:546: auto bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessFlags) const::'lambda'<typename ...$T>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...)::operator()<String>(int, FormatStringHelperImpl<String, std::type_identity<$T>::type...>, $T&&...) const @ 0x0000000019574851
7. /build/src/Access/ContextAccess.cpp:586: bool DB::ContextAccess::checkAccessImplHelper<true, false>(DB::AccessFlags) const @ 0x0000000019572e73
8. /build/src/Access/ContextAccess.cpp:651: bool DB::ContextAccess::checkAccessImpl<true, false>(DB::AccessFlags const&) const @ 0x000000001953f982
9. /build/src/Access/ContextAccess.cpp:724: DB::ContextAccess::checkAccess(DB::AccessFlags const&) const @ 0x000000001953adfd
10. /build/src/Interpreters/Context.cpp:1398: void DB::Context::checkAccessImpl<DB::AccessFlags>(DB::AccessFlags const&) const @ 0x000000001a56f756
11. /build/src/Interpreters/Context.cpp:1401: DB::Context::checkAccess(DB::AccessFlags const&) const @ 0x000000001a544fbd
12. /build/src/Interpreters/InterpreterSystemQuery.cpp:393: DB::InterpreterSystemQuery::execute() @ 0x000000001bbee6c8
13. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
14. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
15. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
16. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
22. ? @ 0x00007f3c91188ac3
23. ? @ 0x00007f3c9121a850

2024.11.05 19:49:50.199416 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/xus/oqhftgbmrddkrnmlpqxfcizthhqji, total size 234881024, count 234881024, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8c2bfbc0-c502-4ad2-899d-ded016c1a8f6, upload has finished false.
2024.11.05 19:49:50.199945 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/xus/oqhftgbmrddkrnmlpqxfcizthhqji, total size 234881024, count 234881024, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8c2bfbc0-c502-4ad2-899d-ded016c1a8f6, upload has finished false
2024.11.05 19:49:50.275589 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/xus/oqhftgbmrddkrnmlpqxfcizthhqji, total size 234881024, count 234881024, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8c2bfbc0-c502-4ad2-899d-ded016c1a8f6, upload has finished false
2024.11.05 19:49:50.317018 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 49 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:50.482534 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:51.626100 [ 5795 ] {50a6bcb8-72cc-4f25-8721-4fc1000bdd68} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=5): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46942) (comment: 00032_aggregate_key64.sql) (in query: SELECT SearchEngineID AS k1, count() AS c FROM test.hits GROUP BY k1 ORDER BY c DESC, k1 LIMIT 10 ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:49:53.345597 [ 5215 ] {7b00d05e-c02e-481b-bd9e-91bf77af356c} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:56140) (in query: select v.`Tuple(a UInt32, b UInt32)`.b from test order by id;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:49:53.410395 [ 5215 ] {7b00d05e-c02e-481b-bd9e-91bf77af356c} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:49:54.162845 [ 2941 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:54.197759 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 3 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:54.383185 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_76_15}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:43: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce60f4
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
24. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
25. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
26. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
27. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
28. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:54.754900 [ 5795 ] {50a6bcb8-72cc-4f25-8721-4fc1000bdd68} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=5): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:49:56.149797 [ 3641 ] {} <Error> void DB::SystemLog<DB::PartLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::PartLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:56.227068 [ 2927 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:49:57.235682 [ 2928 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:00.620860 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 11 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:01.697726 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:02.457608 [ 2935 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:04.513003 [ 2932 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:09.062637 [ 5798 ] {58f47335-de0b-4845-a3cd-7c9263298be0} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:44168) (in query: select v.String from test order by id;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:50:09.097222 [ 5798 ] {58f47335-de0b-4845-a3cd-7c9263298be0} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:50:09.629998 [ 2939 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:10.641500 [ 2941 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:11.622511 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/jun/porzunvporszkkruzawyfrawzsnct, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 07c5f535-45ed-4907-aaa5-7f4549c01dcc, upload has finished false.
2024.11.05 19:50:11.623224 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/jun/porzunvporszkkruzawyfrawzsnct, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 07c5f535-45ed-4907-aaa5-7f4549c01dcc, upload has finished false
2024.11.05 19:50:11.675157 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/jun/porzunvporszkkruzawyfrawzsnct, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 07c5f535-45ed-4907-aaa5-7f4549c01dcc, upload has finished false
2024.11.05 19:50:11.791970 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 32 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:14.338717 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 1 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:14.788621 [ 2930 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:19.401724 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:21.059317 [ 2932 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:21.800782 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {62ec4990-945b-4d8e-9889-ca33c67862f8::202411_1_81_16}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:25.726907 [ 5798 ] {32436bd3-e133-4cf9-8c3c-69198db43d42} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=8): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:58296) (comment: 00050_min_max.sql) (in query: SELECT CounterID, min(WatchID), max(WatchID) FROM test.hits GROUP BY CounterID ORDER BY count() DESC LIMIT 20;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:50:25.743134 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:26.479555 [ 5798 ] {32436bd3-e133-4cf9-8c3c-69198db43d42} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=8): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:50:27.195419 [ 2941 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:29.908850 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/ypr/bqciolvxrffnsulxpwkwugejaecjx, total size 184549376, count 184549376, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 1847278e-ee80-46d5-beca-c4ee6179bdc5, upload has finished false.
2024.11.05 19:50:29.913709 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/ypr/bqciolvxrffnsulxpwkwugejaecjx, total size 184549376, count 184549376, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 1847278e-ee80-46d5-beca-c4ee6179bdc5, upload has finished false
2024.11.05 19:50:30.006093 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/ypr/bqciolvxrffnsulxpwkwugejaecjx, total size 184549376, count 184549376, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 1847278e-ee80-46d5-beca-c4ee6179bdc5, upload has finished false
2024.11.05 19:50:30.087207 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 39 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:31.409415 [ 3810 ] {03f114c4-2f63-45a5-ac84-d7ae8ff6f1db} <Error> executeQuery: Code: 42. DB::Exception: Number of arguments for function isConstant doesn't match: passed 0, should be 1: In scope SELECT isConstant(). (NUMBER_OF_ARGUMENTS_DOESNT_MATCH) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:58304) (comment: 01118_is_constant.sql) (in query: select isConstant();), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, String&&, unsigned long&, unsigned long&) @ 0x0000000016b7ad68
6. /build/src/Functions/IFunction.cpp:431: DB::IFunctionOverloadResolver::checkNumberOfArguments(unsigned long) const @ 0x000000001906536f
7. /build/src/Functions/IFunction.cpp:497: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065836
8. /build/src/Functions/IFunction.cpp:478: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x00000000190657d5
9. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
10. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
11. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
12. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221
13. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6901: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0ab9c7
14. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8173: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d543
15. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
16. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
17. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
19. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
20. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
21. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
28. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
29. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
30. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
31. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660

2024.11.05 19:50:31.422265 [ 3810 ] {03f114c4-2f63-45a5-ac84-d7ae8ff6f1db} <Error> TCPHandler: Code: 42. DB::Exception: Number of arguments for function isConstant doesn't match: passed 0, should be 1: In scope SELECT isConstant(). (NUMBER_OF_ARGUMENTS_DOESNT_MATCH), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, String&&, unsigned long&, unsigned long&) @ 0x0000000016b7ad68
6. /build/src/Functions/IFunction.cpp:431: DB::IFunctionOverloadResolver::checkNumberOfArguments(unsigned long) const @ 0x000000001906536f
7. /build/src/Functions/IFunction.cpp:497: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065836
8. /build/src/Functions/IFunction.cpp:478: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x00000000190657d5
9. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
10. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
11. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
12. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221
13. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6901: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0ab9c7
14. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8173: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d543
15. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
16. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
17. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
19. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
20. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
21. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
28. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
29. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
30. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
31. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660

2024.11.05 19:50:31.570074 [ 3810 ] {af4faa7f-bc3a-4fda-a598-22464fe7ad24} <Error> executeQuery: Code: 42. DB::Exception: Number of arguments for function isConstant doesn't match: passed 2, should be 1: In scope SELECT isConstant(1, 2). (NUMBER_OF_ARGUMENTS_DOESNT_MATCH) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:58304) (comment: 01118_is_constant.sql) (in query: select isConstant(1, 2);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, String&&, unsigned long&, unsigned long&) @ 0x0000000016b7ad68
6. /build/src/Functions/IFunction.cpp:431: DB::IFunctionOverloadResolver::checkNumberOfArguments(unsigned long) const @ 0x000000001906536f
7. /build/src/Functions/IFunction.cpp:497: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065836
8. /build/src/Functions/IFunction.cpp:478: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x00000000190657d5
9. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
10. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
11. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
12. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221
13. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6901: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0ab9c7
14. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8173: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d543
15. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
16. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
17. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
19. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
20. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
21. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
28. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
29. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
30. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
31. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660

2024.11.05 19:50:31.592895 [ 3810 ] {af4faa7f-bc3a-4fda-a598-22464fe7ad24} <Error> TCPHandler: Code: 42. DB::Exception: Number of arguments for function isConstant doesn't match: passed 2, should be 1: In scope SELECT isConstant(1, 2). (NUMBER_OF_ARGUMENTS_DOESNT_MATCH), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, unsigned long&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long&>::type>, String&&, unsigned long&, unsigned long&) @ 0x0000000016b7ad68
6. /build/src/Functions/IFunction.cpp:431: DB::IFunctionOverloadResolver::checkNumberOfArguments(unsigned long) const @ 0x000000001906536f
7. /build/src/Functions/IFunction.cpp:497: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065836
8. /build/src/Functions/IFunction.cpp:478: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x00000000190657d5
9. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
10. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
11. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
12. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221
13. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6901: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0ab9c7
14. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8173: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d543
15. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
16. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
17. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
19. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
20. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
21. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
28. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
29. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
30. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
31. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660

2024.11.05 19:50:32.841692 [ 5793 ] {ff2a9ed7-c159-4af3-8439-4663538146ce} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=3, jobs=1). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:49676) (comment: 00164_quantileBfloat16.sql) (in query: SELECT CounterID AS k, quantileBFloat16(0.5)(ResolutionWidth) FROM test.hits GROUP BY k ORDER BY count() DESC, CounterID LIMIT 10;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:50:32.934190 [ 5793 ] {ff2a9ed7-c159-4af3-8439-4663538146ce} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=3, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:50:33.413427 [ 2931 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:36.485555 [ 2934 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:37.096581 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:40.801139 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:41.651338 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:45.418242 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:45.500367 [ 3810 ] {1a7d585b-a0b4-4a0f-b5b0-c1e6e6044669} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:49702) (comment: 00152_insert_different_granularity.sql) (in query: INSERT INTO fixed_granularity_table SELECT * FROM test.hits LIMIT 10;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:50:45.672136 [ 3810 ] {1a7d585b-a0b4-4a0f-b5b0-c1e6e6044669} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:50:45.750106 [ 3641 ] {} <Error> void DB::SystemLog<DB::PartLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::PartLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:45.797842 [ 2927 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:47.023895 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:47.623175 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/jmk/chccaunngvvxvyfqzroeouguzoqkf, total size 251658240, count 251658240, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id d358a3c2-0bb9-4378-a401-aa420ddcbe11, upload has finished false.
2024.11.05 19:50:47.624426 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/jmk/chccaunngvvxvyfqzroeouguzoqkf, total size 251658240, count 251658240, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id d358a3c2-0bb9-4378-a401-aa420ddcbe11, upload has finished false
2024.11.05 19:50:47.665780 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/jmk/chccaunngvvxvyfqzroeouguzoqkf, total size 251658240, count 251658240, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id d358a3c2-0bb9-4378-a401-aa420ddcbe11, upload has finished false
2024.11.05 19:50:47.730626 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 52 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:47.858700 [ 2930 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:48.892030 [ 2929 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:48.986292 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:50.570206 [ 5793 ] {6bf6a2e5-22c5-41e9-9a8d-873600a0eac1} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:59428) (in query: select v from test order by id;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeReadTask.cpp:44: DB::MergeTreeReadTask::createReaders(std::shared_ptr<DB::MergeTreeReadTaskInfo const> const&, DB::MergeTreeReadTask::Extras const&, DB::MarkRanges const&)::$_0::operator()(DB::NamesAndTypesList const&) const @ 0x000000001ce98fa7
29. /build/src/Storages/MergeTree/MergeTreeReadTask.cpp:57: DB::MergeTreeReadTask::createReaders(std::shared_ptr<DB::MergeTreeReadTaskInfo const> const&, DB::MergeTreeReadTask::Extras const&, DB::MarkRanges const&) @ 0x000000001ce98ac6
30. /build/src/Storages/MergeTree/MergeTreeReadPoolBase.cpp:160: DB::MergeTreeReadPoolBase::createTask(std::shared_ptr<DB::MergeTreeReadTaskInfo const>, DB::MarkRanges, DB::MergeTreeReadTask*) const @ 0x000000001ddaaeea
31. /build/src/Storages/MergeTree/MergeTreeReadPoolInOrder.cpp:70: DB::MergeTreeReadPoolInOrder::getTask(unsigned long, DB::MergeTreeReadTask*) @ 0x000000001ddecc19

2024.11.05 19:50:50.599278 [ 5793 ] {6bf6a2e5-22c5-41e9-9a8d-873600a0eac1} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeReadTask.cpp:44: DB::MergeTreeReadTask::createReaders(std::shared_ptr<DB::MergeTreeReadTaskInfo const> const&, DB::MergeTreeReadTask::Extras const&, DB::MarkRanges const&)::$_0::operator()(DB::NamesAndTypesList const&) const @ 0x000000001ce98fa7
29. /build/src/Storages/MergeTree/MergeTreeReadTask.cpp:57: DB::MergeTreeReadTask::createReaders(std::shared_ptr<DB::MergeTreeReadTaskInfo const> const&, DB::MergeTreeReadTask::Extras const&, DB::MarkRanges const&) @ 0x000000001ce98ac6
30. /build/src/Storages/MergeTree/MergeTreeReadPoolBase.cpp:160: DB::MergeTreeReadPoolBase::createTask(std::shared_ptr<DB::MergeTreeReadTaskInfo const>, DB::MarkRanges, DB::MergeTreeReadTask*) const @ 0x000000001ddaaeea
31. /build/src/Storages/MergeTree/MergeTreeReadPoolInOrder.cpp:70: DB::MergeTreeReadPoolInOrder::getTask(unsigned long, DB::MergeTreeReadTask*) @ 0x000000001ddecc19

2024.11.05 19:50:55.685960 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/rzv/sghmnosdcabejahjozcoiqypshwot, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 82c06a39-7482-4f4f-b830-09132cf2660e, upload has finished false.
2024.11.05 19:50:55.692355 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/rzv/sghmnosdcabejahjozcoiqypshwot, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 82c06a39-7482-4f4f-b830-09132cf2660e, upload has finished false
2024.11.05 19:50:55.776555 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/rzv/sghmnosdcabejahjozcoiqypshwot, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 82c06a39-7482-4f4f-b830-09132cf2660e, upload has finished false
2024.11.05 19:50:55.801941 [ 3800 ] {31f6fd23-cd97-4254-806f-ff69d7f6047b} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48410) (comment: 00150_quantiles_timing_precision.sql) (in query: SELECT CounterID, quantileTiming(0.5)(SendTiming) AS qt, least(30000, quantileExact(0.5)(SendTiming)) AS qe, count() AS c, round(abs(qt - qe) / greatest(qt, qe) AS diff, 3) AS rounded_diff FROM test.hits WHERE SendTiming != -1 GROUP BY CounterID HAVING diff != 0 ORDER BY diff DESC SETTINGS optimize_aggregation_in_order = 1;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:50:55.842998 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 20 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:50:56.043544 [ 3800 ] {31f6fd23-cd97-4254-806f-ff69d7f6047b} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:50:57.648431 [ 3584 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key s3/lol/vigikvjoixggfepzrajuojyodpejn, total size 122843301, count 122843301, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id 9a1a778b-9145-470b-b96e-2a7cb9fa71a2, upload has finished false.
2024.11.05 19:50:57.665025 [ 3584 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key s3/lol/vigikvjoixggfepzrajuojyodpejn, total size 122843301, count 122843301, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id 9a1a778b-9145-470b-b96e-2a7cb9fa71a2, upload has finished false
2024.11.05 19:50:57.771731 [ 3584 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key s3/lol/vigikvjoixggfepzrajuojyodpejn, total size 122843301, count 122843301, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id 9a1a778b-9145-470b-b96e-2a7cb9fa71a2, upload has finished false
2024.11.05 19:50:57.801411 [ 3584 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key s3/uzp/xhbhwjdrmfeovlyyhklytieoyobcy, total size 44068035, count 44068035, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id 26c15f55-5449-4674-bcce-b6fc082a003c, upload has finished false.
2024.11.05 19:50:57.802634 [ 3584 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key s3/uzp/xhbhwjdrmfeovlyyhklytieoyobcy, total size 44068035, count 44068035, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id 26c15f55-5449-4674-bcce-b6fc082a003c, upload has finished false
2024.11.05 19:50:57.875559 [ 3584 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key s3/uzp/xhbhwjdrmfeovlyyhklytieoyobcy, total size 44068035, count 44068035, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id 26c15f55-5449-4674-bcce-b6fc082a003c, upload has finished false
2024.11.05 19:50:58.209319 [ 3584 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key s3/lxm/oeckwrgyscklpirnhcrjjmmfzptti, total size 49586881, count 49586881, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id 8124b082-a9be-4b51-8e30-d2def2d3fb5c, upload has finished false.
2024.11.05 19:50:58.211882 [ 3584 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key s3/lxm/oeckwrgyscklpirnhcrjjmmfzptti, total size 49586881, count 49586881, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id 8124b082-a9be-4b51-8e30-d2def2d3fb5c, upload has finished false
2024.11.05 19:50:58.252965 [ 3584 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key s3/lxm/oeckwrgyscklpirnhcrjjmmfzptti, total size 49586881, count 49586881, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id 8124b082-a9be-4b51-8e30-d2def2d3fb5c, upload has finished false
2024.11.05 19:50:59.228939 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:43: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce60f4
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
23. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
24. /build/src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp:69: DB::MergedColumnOnlyOutputStream::fillChecksums(std::shared_ptr<DB::IMergeTreeDataPart>&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3fe1a
25. /build/src/Storages/MergeTree/MergeTask.cpp:657: DB::MergeTask::VerticalMergeStage::finalizeVerticalMergeForOneColumn() const @ 0x000000001cb6fabd
26. /build/src/Storages/MergeTree/MergeTask.cpp:877: DB::MergeTask::VerticalMergeStage::executeVerticalMergeForAllColumns() const @ 0x000000001cb72f6e
27. /build/src/Storages/MergeTree/MergeTask.cpp:829: DB::MergeTask::VerticalMergeStage::execute() @ 0x000000001cb72cb8
28. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
29. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:00.376923 [ 2941 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:02.409333 [ 2927 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:02.497100 [ 3580 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/yzw/nvxlzmxznrnccqvwvrpcrdieesbwm, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id cfdaa17e-4161-4b54-93cc-77fcfa1d5680, upload has finished false.
2024.11.05 19:51:02.497878 [ 3580 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/yzw/nvxlzmxznrnccqvwvrpcrdieesbwm, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id cfdaa17e-4161-4b54-93cc-77fcfa1d5680, upload has finished false
2024.11.05 19:51:02.530945 [ 3580 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/yzw/nvxlzmxznrnccqvwvrpcrdieesbwm, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id cfdaa17e-4161-4b54-93cc-77fcfa1d5680, upload has finished false
2024.11.05 19:51:02.631623 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 12 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:04.462553 [ 2930 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:04.902282 [ 5798 ] {90bf3760-532d-4348-8b3a-dbeb08b76931} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=25, jobs=25). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:45172) (comment: 00173_group_by_use_nulls.sql) (in query: SELECT CounterID AS k, quantileBFloat16(0.5)(ResolutionWidth) FROM remote('127.0.0.{1,2}', test, hits) GROUP BY k ORDER BY count() DESC, CounterID ASC LIMIT 10 SETTINGS group_by_use_nulls = 1;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:51:05.140168 [ 5798 ] {90bf3760-532d-4348-8b3a-dbeb08b76931} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=25, jobs=25). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:51:05.463233 [ 3800 ] {1c19545b-31c6-4f61-bedb-eaf20829e1a7} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59592) (comment: 00173_group_by_use_nulls.sql) (in query: SELECT `__table1`.`CounterID` AS `k`, quantileBFloat16(0.5)(`__table1`.`ResolutionWidth`) AS `quantileBFloat16(0.5)(ResolutionWidth)` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`CounterID` ORDER BY count() DESC, `__table1`.`CounterID` ASC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:51:05.558489 [ 3800 ] {1c19545b-31c6-4f61-bedb-eaf20829e1a7} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:51:05.561858 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:05.565553 [ 3800 ] {1c19545b-31c6-4f61-bedb-eaf20829e1a7} <Warning> TCPHandler: Client has gone away.
2024.11.05 19:51:05.564716 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column UserID): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_1_1_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 1 with max_rows_to_read = 6658): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:06.506018 [ 2931 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:10.098041 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=1): (while reading column Sign): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_2_2_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 0 with max_rows_to_read = 6592): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:10.872531 [ 5793 ] {1a4cad69-9a71-44f7-b266-5ea075706ef4} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:45204) (comment: 01720_country_intersection.sh) (in query: CREATE TABLE country_polygons (`name` String, `p` Array(Array(Tuple(Float64, Float64)))) ENGINE = MergeTree ORDER BY tuple() SETTINGS index_granularity = 1670, min_bytes_for_wide_part = 0, ratio_of_defaults_for_sparse_serialization = 1., replace_long_file_name_to_hash = true, max_file_name_length = 0, min_bytes_for_full_part_storage = 536870912, compact_parts_max_bytes_to_buffer = 417991500, compact_parts_max_granules_to_buffer = 256, compact_parts_merge_max_bytes_to_prefetch_part = 25538600, merge_max_block_size = 2461, old_parts_lifetime = 425., prefer_fetch_merged_part_size_threshold = 10737418240, vertical_merge_algorithm_min_rows_to_activate = 1000000, vertical_merge_algorithm_min_columns_to_activate = 1, min_merge_bytes_to_use_direct_io = 1, index_granularity_bytes = 15849091, concurrent_part_removal_threshold = 80, allow_vertical_merges_from_compact_to_wide_parts = true, cache_populated_by_fetch = false, marks_compress_block_size = 22930, primary_key_compress_block_size = 9132), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/MergeTree/MergeTreeData.cpp:1703: DB::MergeTreeData::loadDataParts(bool, std::optional<std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>>) @ 0x000000001cbc7077
11. /build/src/Storages/StorageMergeTree.cpp:120: DB::StorageMergeTree::StorageMergeTree(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata const&, DB::LoadingStrictnessLevel, std::shared_ptr<DB::Context>, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>) @ 0x000000001d0eba14
12. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageMergeTree* std::construct_at[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::StorageMergeTree*>(DB::StorageMergeTree*, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eb3b7
13. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageMergeTree>>::construct[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void, void>(std::allocator<DB::StorageMergeTree>&, DB::StorageMergeTree*, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eb0c7
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageMergeTree, std::allocator<DB::StorageMergeTree>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>>(std::allocator<DB::StorageMergeTree>, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0ead75
15. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageMergeTree> std::allocate_shared[abi:v15000]<DB::StorageMergeTree, std::allocator<DB::StorageMergeTree>, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void>(std::allocator<DB::StorageMergeTree> const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eaba7
16. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageMergeTree> std::make_shared[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void>(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0e7d6d
17. /build/src/Storages/MergeTree/registerStorageMergeTree.cpp:741: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000001d0e58aa
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000193086fd
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000193086bd
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001930867d
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019308645
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
24. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
25. /build/src/Interpreters/InterpreterCreateQuery.cpp:1493: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x000000001b01ec0c
26. /build/src/Interpreters/InterpreterCreateQuery.cpp:1313: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b018b66
27. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
28. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
29. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
30. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
31. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5

2024.11.05 19:51:10.932936 [ 5793 ] {1a4cad69-9a71-44f7-b266-5ea075706ef4} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/MergeTree/MergeTreeData.cpp:1703: DB::MergeTreeData::loadDataParts(bool, std::optional<std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>>) @ 0x000000001cbc7077
11. /build/src/Storages/StorageMergeTree.cpp:120: DB::StorageMergeTree::StorageMergeTree(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata const&, DB::LoadingStrictnessLevel, std::shared_ptr<DB::Context>, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>) @ 0x000000001d0eba14
12. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageMergeTree* std::construct_at[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::StorageMergeTree*>(DB::StorageMergeTree*, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eb3b7
13. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageMergeTree>>::construct[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void, void>(std::allocator<DB::StorageMergeTree>&, DB::StorageMergeTree*, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eb0c7
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageMergeTree, std::allocator<DB::StorageMergeTree>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>>(std::allocator<DB::StorageMergeTree>, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0ead75
15. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageMergeTree> std::allocate_shared[abi:v15000]<DB::StorageMergeTree, std::allocator<DB::StorageMergeTree>, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void>(std::allocator<DB::StorageMergeTree> const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eaba7
16. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageMergeTree> std::make_shared[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void>(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0e7d6d
17. /build/src/Storages/MergeTree/registerStorageMergeTree.cpp:741: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000001d0e58aa
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000193086fd
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000193086bd
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001930867d
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019308645
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
24. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
25. /build/src/Interpreters/InterpreterCreateQuery.cpp:1493: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x000000001b01ec0c
26. /build/src/Interpreters/InterpreterCreateQuery.cpp:1313: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b018b66
27. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
28. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
29. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
30. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
31. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5

2024.11.05 19:51:10.964506 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {d830f0a7-c4af-4441-961e-a7e3a729e583::202411_1_91_18}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:12.106494 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:12.595970 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:12.755149 [ 2936 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:12.829472 [ 5798 ] {bec01098-647b-4a37-aedb-bce222f1b2db} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:45214) (comment: 00078_group_by_arrays.sql) (in query: SELECT GoalsReached AS k, count() AS c FROM test.hits GROUP BY k ORDER BY c DESC LIMIT 10;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:51:12.866833 [ 5798 ] {bec01098-647b-4a37-aedb-bce222f1b2db} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:51:13.756629 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:15.082077 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/fuc/sdtlbmtyxsnxfdvxdpzojpvnbpmcx, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id a4c9114d-59e5-4330-a6b2-e08963e20019, upload has finished false.
2024.11.05 19:51:15.086761 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/fuc/sdtlbmtyxsnxfdvxdpzojpvnbpmcx, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id a4c9114d-59e5-4330-a6b2-e08963e20019, upload has finished false
2024.11.05 19:51:15.148734 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/fuc/sdtlbmtyxsnxfdvxdpzojpvnbpmcx, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id a4c9114d-59e5-4330-a6b2-e08963e20019, upload has finished false
2024.11.05 19:51:15.244264 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 28 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:15.260056 [ 7051 ] {1e61119d-db39-4d08-8d49-bab9b970c4fc} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:51072) (comment: 00173_group_by_use_nulls.sql) (in query: SELECT `__table1`.`CounterID` AS `k`, quantileBFloat16(0.5)(`__table1`.`ResolutionWidth`) AS `quantileBFloat16(0.5)(ResolutionWidth)` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`CounterID` ORDER BY count() DESC, `__table1`.`CounterID` ASC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:51:15.906827 [ 5793 ] {81a75090-3559-415f-9e5d-a236967b7b6e} <Error> executeQuery: Code: 60. DB::Exception: Table test_15.country_polygons does not exist. (UNKNOWN_TABLE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48826) (comment: 01720_country_intersection.sh) (in query: insert into country_polygons format TSV), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000a8f2fcc
6. /build/src/Databases/IDatabase.cpp:37: DB::IDatabase::getTable(String const&, std::shared_ptr<DB::Context const>) const @ 0x000000001a146372
7. /build/src/Interpreters/DatabaseCatalog.cpp:429: DB::DatabaseCatalog::getTableImpl(DB::StorageID const&, std::shared_ptr<DB::Context const>, std::optional<DB::Exception>*) const @ 0x000000001a670796
8. /build/src/Interpreters/DatabaseCatalog.cpp:952: DB::DatabaseCatalog::getTable(DB::StorageID const&, std::shared_ptr<DB::Context const>) const @ 0x000000001a675ae3
9. /build/src/Interpreters/InterpreterInsertQuery.cpp:123: DB::InterpreterInsertQuery::getTable(DB::ASTInsertQuery&) @ 0x000000001b429f12
10. /build/src/Interpreters/InterpreterInsertQuery.cpp:406: DB::InterpreterInsertQuery::execute() @ 0x000000001b42c384
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:51:15.934824 [ 5793 ] {81a75090-3559-415f-9e5d-a236967b7b6e} <Error> TCPHandler: Code: 60. DB::Exception: Table test_15.country_polygons does not exist. (UNKNOWN_TABLE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000a8f2fcc
6. /build/src/Databases/IDatabase.cpp:37: DB::IDatabase::getTable(String const&, std::shared_ptr<DB::Context const>) const @ 0x000000001a146372
7. /build/src/Interpreters/DatabaseCatalog.cpp:429: DB::DatabaseCatalog::getTableImpl(DB::StorageID const&, std::shared_ptr<DB::Context const>, std::optional<DB::Exception>*) const @ 0x000000001a670796
8. /build/src/Interpreters/DatabaseCatalog.cpp:952: DB::DatabaseCatalog::getTable(DB::StorageID const&, std::shared_ptr<DB::Context const>) const @ 0x000000001a675ae3
9. /build/src/Interpreters/InterpreterInsertQuery.cpp:123: DB::InterpreterInsertQuery::getTable(DB::ASTInsertQuery&) @ 0x000000001b429f12
10. /build/src/Interpreters/InterpreterInsertQuery.cpp:406: DB::InterpreterInsertQuery::execute() @ 0x000000001b42c384
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
13. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:51:16.019571 [ 3810 ] {e9ae3c34-d21a-40dc-8e02-0aabe3870ca6} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48818) (in query: insert into test select number + 3, number % 2 ? NULL : number from numbers(3);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d

2024.11.05 19:51:16.066386 [ 3810 ] {e9ae3c34-d21a-40dc-8e02-0aabe3870ca6} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d

2024.11.05 19:51:16.372625 [ 7051 ] {1e61119d-db39-4d08-8d49-bab9b970c4fc} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:51072). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:169: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e9244
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Server/TCPHandler.cpp:2253: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001d487855
9. /build/src/Server/TCPHandler.cpp:2330: DB::TCPHandler::sendLogs() @ 0x000000001d47c5ff
10. /build/src/Server/TCPHandler.cpp:693: DB::TCPHandler::runImpl() @ 0x000000001d471ac2
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:16.402062 [ 7051 ] {1e61119d-db39-4d08-8d49-bab9b970c4fc} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:51:16.420311 [ 7051 ] {1e61119d-db39-4d08-8d49-bab9b970c4fc} <Warning> TCPHandler: Client has gone away.
2024.11.05 19:51:16.903530 [ 2941 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:19.463443 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 2 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:20.985133 [ 3800 ] {0ba267cf-47db-4e2e-a393-11346c8e33ec} <Error> executeQuery: Code: 60. DB::Exception: Unknown table expression identifier 'country_polygons' in scope SELECT first.p AS a, second.p AS b, first.name AS c, second.name AS d FROM country_polygons AS first CROSS JOIN country_polygons AS second LIMIT 100. (UNKNOWN_TABLE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48842) (comment: 01720_country_intersection.sh) (in query: SELECT c, d, polygonsIntersectionSpherical(a, b) FROM (SELECT first.p AS a, second.p AS b, first.name AS c, second.name AS d FROM country_polygons AS first CROSS JOIN country_polygons AS second LIMIT 100) format TSV), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6970: DB::(anonymous namespace)::QueryAnalyzer::initializeQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0aa552
7. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8159: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4a3
8. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6519: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a3062
9. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:7911: DB::(anonymous namespace)::QueryAnalyzer::resolveQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, DB::(anonymous namespace)::QueryExpressionsAliasVisitor&) @ 0x000000001b0ab48f
10. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8164: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4d6
11. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
12. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
13. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
14. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
15. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
16. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
24. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
25. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
26. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
27. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
28. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
29. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
30. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
31. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194

2024.11.05 19:51:21.011267 [ 3800 ] {0ba267cf-47db-4e2e-a393-11346c8e33ec} <Error> TCPHandler: Code: 60. DB::Exception: Unknown table expression identifier 'country_polygons' in scope SELECT first.p AS a, second.p AS b, first.name AS c, second.name AS d FROM country_polygons AS first CROSS JOIN country_polygons AS second LIMIT 100. (UNKNOWN_TABLE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6970: DB::(anonymous namespace)::QueryAnalyzer::initializeQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0aa552
7. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8159: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4a3
8. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6519: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a3062
9. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:7911: DB::(anonymous namespace)::QueryAnalyzer::resolveQueryJoinTreeNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, DB::(anonymous namespace)::QueryExpressionsAliasVisitor&) @ 0x000000001b0ab48f
10. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8164: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b09d4d6
11. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:1150: DB::(anonymous namespace)::QueryAnalyzer::resolve(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::IQueryTreeNode> const&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b774
12. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:8408: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000001b09b433
13. /build/src/Analyzer/QueryTreePassManager.cpp:186: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001b08cba6
14. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:142: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x000000001b5085ac
15. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001b5071e6
16. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&) @ 0x000000001b50a5b7
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:270: DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0::operator()(DB::InterpreterFactory::Arguments const&) const @ 0x000000001b509a4c
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001b5099fd
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001b50999d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001b50995d
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001b509925
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001b42931e
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001b42489d
24. /build/src/Interpreters/InterpreterFactory.cpp:355: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001b423f2d
25. /build/src/Interpreters/executeQuery.cpp:1133: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb24b0
26. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
27. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
28. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
29. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
30. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
31. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194

2024.11.05 19:51:23.075756 [ 2931 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:25.544873 [ 5795 ] {9308f255-49ba-47b5-82ad-dc2aa277b177} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:49734) (comment: 02503_cache_on_write_with_small_segment_size.sh) (in query:  select count() from system.filesystem_cache_log where query_id = 'vfSSStDv' AND read_type != 'READ_FROM_CACHE'; ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:51:25.551251 [ 5795 ] {9308f255-49ba-47b5-82ad-dc2aa277b177} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:51:28.273032 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/bff/ijrartceanebntixiuhokhhhzawqc, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id cc718d48-e72c-49d0-9a63-6880de881b25, upload has finished false.
2024.11.05 19:51:28.273780 [ 2937 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:28.275224 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/bff/ijrartceanebntixiuhokhhhzawqc, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id cc718d48-e72c-49d0-9a63-6880de881b25, upload has finished false
2024.11.05 19:51:28.333310 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/bff/ijrartceanebntixiuhokhhhzawqc, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id cc718d48-e72c-49d0-9a63-6880de881b25, upload has finished false
2024.11.05 19:51:28.369684 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 19 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:28.703490 [ 3810 ] {bda2dc14-4308-4797-9fac-d6504c33fd2c} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=8, jobs=8). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48828) (comment: 00044_any_left_join_string.sql) (in query: SELECT domain, hits, visits FROM ( SELECT domain(URL) AS domain, count() AS hits FROM test.hits GROUP BY domain ) ANY LEFT JOIN ( SELECT domain(StartURL) AS domain, sum(Sign) AS visits FROM test.visits GROUP BY domain ) USING domain ORDER BY hits DESC LIMIT 10 SETTINGS joined_subquery_requires_alias = 0;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:51:28.789240 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:28.979280 [ 5793 ] {7bec4a19-f23b-4c7a-9373-e9880919c875} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=8, jobs=8). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:49762) (comment: 02503_cache_on_write_with_small_segment_size.sh) (in query: select count() from ttt;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:51:28.986085 [ 5793 ] {7bec4a19-f23b-4c7a-9373-e9880919c875} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=8, jobs=8). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:51:29.283140 [ 2936 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:30.630128 [ 3810 ] {bda2dc14-4308-4797-9fac-d6504c33fd2c} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=8, jobs=8). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:51:31.360351 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:33.587140 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:33.604370 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_87_17}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=10). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:33.586707 [ 3810 ] {ae0845b0-81f1-4f6b-97de-09e77e17d101} <Error> executeQuery: Code: 36. DB::Exception: Cannot load time zone /abc: In scope SELECT toDateTime(0, '/abc'). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48150) (comment: 02505_forbid_paths_in_datetime_timezone.sql) (in query: select toDateTime(0, '/abc');), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:51:33.681948 [ 3810 ] {ae0845b0-81f1-4f6b-97de-09e77e17d101} <Error> TCPHandler: Code: 36. DB::Exception: Cannot load time zone /abc: In scope SELECT toDateTime(0, '/abc'). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:51:33.761379 [ 3810 ] {be9024a6-1427-4fb1-8acd-22221965b67e} <Error> executeQuery: Code: 36. DB::Exception: Cannot load time zone ./abc: In scope SELECT toDateTime(0, './abc'). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48150) (comment: 02505_forbid_paths_in_datetime_timezone.sql) (in query: select toDateTime(0, './abc');), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:51:33.771560 [ 3810 ] {be9024a6-1427-4fb1-8acd-22221965b67e} <Error> TCPHandler: Code: 36. DB::Exception: Cannot load time zone ./abc: In scope SELECT toDateTime(0, './abc'). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:51:33.889597 [ 3810 ] {6cfcc4bb-2679-4f06-9fb7-94cd54d343b0} <Error> executeQuery: Code: 36. DB::Exception: Cannot load time zone ../abc: In scope SELECT toDateTime(0, '../abc'). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48150) (comment: 02505_forbid_paths_in_datetime_timezone.sql) (in query: select toDateTime(0, '../abc');), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:51:33.894138 [ 3810 ] {6cfcc4bb-2679-4f06-9fb7-94cd54d343b0} <Error> TCPHandler: Code: 36. DB::Exception: Cannot load time zone ../abc: In scope SELECT toDateTime(0, '../abc'). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:51:34.039286 [ 3810 ] {62ff3719-935a-4e83-9598-45372e07cd2a} <Error> executeQuery: Code: 36. DB::Exception: Cannot load time zone ~/abc: In scope SELECT toDateTime(0, '~/abc'). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48150) (comment: 02505_forbid_paths_in_datetime_timezone.sql) (in query: select toDateTime(0, '~/abc');), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:51:34.099625 [ 3810 ] {62ff3719-935a-4e83-9598-45372e07cd2a} <Error> TCPHandler: Code: 36. DB::Exception: Cannot load time zone ~/abc: In scope SELECT toDateTime(0, '~/abc'). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:51:34.267882 [ 3810 ] {a7aba751-c4a3-444d-8af6-dee6f9a6ad7b} <Error> executeQuery: Code: 36. DB::Exception: Cannot load time zone abc/../../cba: In scope SELECT toDateTime(0, 'abc/../../cba'). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48150) (comment: 02505_forbid_paths_in_datetime_timezone.sql) (in query: select toDateTime(0, 'abc/../../cba');), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:51:34.278418 [ 3810 ] {a7aba751-c4a3-444d-8af6-dee6f9a6ad7b} <Error> TCPHandler: Code: 36. DB::Exception: Cannot load time zone abc/../../cba: In scope SELECT toDateTime(0, 'abc/../../cba'). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:51:40.895768 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/lnv/cjhtlstphxbjanpobjpvrnbzzinyc, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 932f59e3-e3c3-4aeb-9f63-0d1acb0d83ea, upload has finished false.
2024.11.05 19:51:40.897625 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/lnv/cjhtlstphxbjanpobjpvrnbzzinyc, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 932f59e3-e3c3-4aeb-9f63-0d1acb0d83ea, upload has finished false
2024.11.05 19:51:40.970923 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/lnv/cjhtlstphxbjanpobjpvrnbzzinyc, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 932f59e3-e3c3-4aeb-9f63-0d1acb0d83ea, upload has finished false
2024.11.05 19:51:41.095295 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 31 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:41.609619 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:41.916473 [ 2935 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:42.794706 [ 3810 ] {c12fc3d0-43bc-48e5-91f2-251329b92a12} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=14, jobs=14). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48182) (comment: 00045_uniq_upto.sql) (in query: SELECT RegionID, uniqExact(UserID) AS u1, uniqUpTo(10)(UserID) AS u2 FROM test.visits GROUP BY RegionID HAVING u1 <= 11 AND u1 != u2), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:51:43.022178 [ 5795 ] {41b64e15-0f18-4a8d-9bd4-044c3e27d0f3} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=10, jobs=10). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48204) (comment: 00079_array_join_not_used_joined_column.sql) (in query: SELECT PP.Key1 AS `ym:s:paramsLevel1`, sum(arrayAll(`x_1` -> `x_1`= '', ParsedParams.Key2)) AS `ym:s:visits` FROM test.hits ARRAY JOIN ParsedParams AS `PP` WHERE CounterID = 1704509 GROUP BY `ym:s:paramsLevel1` ORDER BY PP.Key1, `ym:s:visits` LIMIT 0, 100;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:51:44.215156 [ 5795 ] {41b64e15-0f18-4a8d-9bd4-044c3e27d0f3} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=10, jobs=10). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:51:44.235806 [ 3642 ] {} <Error> void DB::SystemLog<DB::MetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::MetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=7). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:44.458898 [ 3810 ] {c12fc3d0-43bc-48e5-91f2-251329b92a12} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=14, jobs=14). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:51:44.982238 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 6 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:46.001033 [ 2936 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:51.658871 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/nji/bwugdbnrotyqxjoopttoziyftupkt, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6dcef29e-198e-4992-aad0-2e64460b6a02, upload has finished false.
2024.11.05 19:51:51.661596 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/nji/bwugdbnrotyqxjoopttoziyftupkt, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6dcef29e-198e-4992-aad0-2e64460b6a02, upload has finished false
2024.11.05 19:51:51.708461 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/nji/bwugdbnrotyqxjoopttoziyftupkt, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6dcef29e-198e-4992-aad0-2e64460b6a02, upload has finished false
2024.11.05 19:51:51.861846 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 17 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:54.215238 [ 3810 ] {3e9cfd47-9881-4078-a70a-701af2bacea5} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=23, jobs=23). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:33206) (comment: 00078_group_by_arrays.sql) (in query: SELECT GoalsReached AS k, count() AS c FROM test.hits GROUP BY k ORDER BY c DESC LIMIT 10;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:51:54.306991 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {c04f5a1b-d832-4de4-96b0-dec4e591ea83::202411_1_81_16}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:55.166768 [ 3810 ] {3e9cfd47-9881-4078-a70a-701af2bacea5} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=23, jobs=23). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:51:57.435135 [ 2933 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:58.585188 [ 2935 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:51:58.726282 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/wus/falzxcjvugsrogtatesiqjdhwkvrv, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b8fe48f4-ecb4-42a1-90b3-b839ba2ad819, upload has finished false.
2024.11.05 19:51:58.730241 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/wus/falzxcjvugsrogtatesiqjdhwkvrv, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b8fe48f4-ecb4-42a1-90b3-b839ba2ad819, upload has finished false
2024.11.05 19:51:58.777480 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/wus/falzxcjvugsrogtatesiqjdhwkvrv, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b8fe48f4-ecb4-42a1-90b3-b839ba2ad819, upload has finished false
2024.11.05 19:51:58.971762 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 12 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:01.666255 [ 2937 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=4, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:02.321081 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 2 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:02.688075 [ 2936 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:03.485392 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:03.606581 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:03.767844 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:05.756085 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 2 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:06.732027 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:08.349384 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 3 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:09.622509 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column Hits): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_1_1_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 0 with max_rows_to_read = 6658): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:09.754726 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:09.898760 [ 2928 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:11.886459 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 1 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:13.877619 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:15.553866 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:15.690218 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 7 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:17.478896 [ 5215 ] {79c8aa7e-9e55-4579-bfff-f2b72e7f4262} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=3): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46088) (comment: 00163_column_oriented_formats.sh) (in query: INSERT INTO 00163_column_oriented SELECT ClientEventTime, MobilePhoneModel, ClientIP6 FROM test.hits ORDER BY ClientEventTime, MobilePhoneModel, ClientIP6 LIMIT 100), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:52:18.902849 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:19.798015 [ 5215 ] {79c8aa7e-9e55-4579-bfff-f2b72e7f4262} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=3): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:52:20.399456 [ 2938 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:22.710558 [ 3611 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Removing temporary directory /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:52:22.711142 [ 3611 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:52:22.988973 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/mhv/llxjxtchfegroqxsxlpjdzneevmqy, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c5fe3190-8096-433b-a6c9-ae0a91465f52, upload has finished false.
2024.11.05 19:52:23.002309 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/mhv/llxjxtchfegroqxsxlpjdzneevmqy, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c5fe3190-8096-433b-a6c9-ae0a91465f52, upload has finished false
2024.11.05 19:52:23.108049 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/mhv/llxjxtchfegroqxsxlpjdzneevmqy, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c5fe3190-8096-433b-a6c9-ae0a91465f52, upload has finished false
2024.11.05 19:52:23.166657 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 16 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:23.862579 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:24.256048 [ 3810 ] {21265ac1-007a-456f-9424-0e6e043c34a2} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=27, jobs=27). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47034) (comment: 00163_column_oriented_formats.sh) (in query: SELECT ClientEventTime from 00163_column_oriented), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:52:24.342937 [ 3810 ] {21265ac1-007a-456f-9424-0e6e043c34a2} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=27, jobs=27). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:52:24.477707 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column n): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:26.430332 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {dbd666d3-2d23-40c6-8136-6ca2be336b4d::202411_1_72_13}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:26.650126 [ 2928 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:26.771368 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_93_18}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:28.692532 [ 3810 ] {1d2bc69c-6d11-450a-80ef-06d35dd84328} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47060) (comment: 00163_column_oriented_formats.sh) (in query: SELECT MobilePhoneModel from 00163_column_oriented), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:52:28.701015 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 6 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:28.706595 [ 2929 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:28.748251 [ 3810 ] {1d2bc69c-6d11-450a-80ef-06d35dd84328} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:52:29.726878 [ 2931 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:30.678760 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:32.458317 [ 3641 ] {} <Error> void DB::SystemLog<DB::PartLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::PartLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:33.508186 [ 3810 ] {d2b67d33-4daa-4aa0-8d2f-eec571765f1f} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=43, jobs=43). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:42098) (comment: 00163_column_oriented_formats.sh) (in query: SELECT ClientIP6 from 00163_column_oriented), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:52:33.677571 [ 3810 ] {d2b67d33-4daa-4aa0-8d2f-eec571765f1f} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=43, jobs=43). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:52:34.018189 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_93_18}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=7). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:45: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce610d
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
24. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
25. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
26. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
27. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
28. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:35.745899 [ 5795 ] {94b69f34-e0e5-4360-b7fd-99132cb75a82} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=3). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47066) (in query: insert into test select number + 6, 'str_' || toString(number) from numbers(3);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:45: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce610d
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375

2024.11.05 19:52:35.780956 [ 5795 ] {94b69f34-e0e5-4360-b7fd-99132cb75a82} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:45: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce610d
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375

2024.11.05 19:52:38.020523 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/lih/ixknvpmovbpsfnedcrgydtpeevpag, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 2163f555-aee6-49b3-a34a-940a542c467f, upload has finished false.
2024.11.05 19:52:38.021376 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/lih/ixknvpmovbpsfnedcrgydtpeevpag, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 2163f555-aee6-49b3-a34a-940a542c467f, upload has finished false
2024.11.05 19:52:38.051850 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/lih/ixknvpmovbpsfnedcrgydtpeevpag, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 2163f555-aee6-49b3-a34a-940a542c467f, upload has finished false
2024.11.05 19:52:38.275593 [ 2940 ] {} <Error> void DB::StorageBuffer::backgroundFlush(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:169: DB::ThreadPoolCallbackRunnerLocal<void, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>, std::function<void ()>>::operator()(std::function<void ()>&&, Priority) @ 0x0000000018f6967a
10. /build/src/Storages/StorageBuffer.cpp:867: DB::StorageBuffer::flushAllBuffers(bool) @ 0x000000001c781af2
11. /build/src/Storages/StorageBuffer.cpp:1035: DB::StorageBuffer::backgroundFlush() @ 0x000000001c78695e
12. /build/src/Storages/StorageBuffer.cpp:168: DB::StorageBuffer::StorageBuffer(DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr<DB::Context const>, unsigned long, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageBuffer::Thresholds const&, DB::StorageID const&, bool)::$_0::operator()() const @ 0x000000001c787c38
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001c787c15
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001c787bf5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001c787bd5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c787b9d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Core/BackgroundSchedulePool.cpp:106: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000019932e9d
20. /build/src/Core/BackgroundSchedulePool.cpp:304: DB::BackgroundSchedulePool::threadFunction() @ 0x0000000019936906
21. /build/src/Core/BackgroundSchedulePool.cpp:170: DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0::operator()() const @ 0x0000000019937818
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199377f5
23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000199377d9
24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&, std::tuple<>&) @ 0x000000001993773d
25. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'()::operator()() @ 0x0000000019937665
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000199375d5
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000199375b5
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x0000000019937595
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019937560
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:38.279406 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:567: DB::WriteBufferFromS3::writePart(DB::WriteBufferFromS3::PartData&&) @ 0x0000000019914d6e
18. /build/src/IO/WriteBufferFromS3.cpp:370: DB::WriteBufferFromS3::writeMultipartUpload() @ 0x0000000019908eb6
19. /build/src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::nextImpl() @ 0x000000001990857e
20. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
21. /build/src/IO/WriteBufferFromFileDecorator.cpp:71: DB::WriteBufferFromFileDecorator::nextImpl() @ 0x000000001a3593ad
22. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
23. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
24. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
25. /build/src/IO/WriteBuffer.h:76: DB::WriteBuffer::nextIfAtEnd() @ 0x000000000a931978
26. /build/src/IO/WriteBuffer.h:91: DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000a9317e0
27. /build/src/Compression/CompressedWriteBuffer.cpp:58: DB::CompressedWriteBuffer::nextImpl() @ 0x0000000019ddd7ed
28. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
29. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
30. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
31. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:350: DB::MergeTreeDataPartWriterWide::getCurrentMarksForColumn(DB::NameAndTypePair const&, std::set<String, std::less<String>, std::allocator<String>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ccf5839
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:38.858528 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {86a06611-889e-455b-b452-b6a6203be276::202411_1_75_15}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:39.006587 [ 3612 ] {} <Warning> system.trace_log (4ff3d7e3-1bf7-4d39-8a23-c88787b766d0): Removing temporary directory /var/lib/clickhouse/disks/s3/store/4ff/4ff3d7e3-1bf7-4d39-8a23-c88787b766d0/tmp_merge_202411_1_93_18/
2024.11.05 19:52:39.009328 [ 3612 ] {} <Warning> system.trace_log (4ff3d7e3-1bf7-4d39-8a23-c88787b766d0): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3/store/4ff/4ff3d7e3-1bf7-4d39-8a23-c88787b766d0/tmp_merge_202411_1_93_18/
2024.11.05 19:52:39.454824 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_93_18}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column event): (while reading from part /var/lib/clickhouse/disks/s3/store/4ff/4ff3d7e3-1bf7-4d39-8a23-c88787b766d0/202411_1_87_17/ in table system.trace_log (4ff3d7e3-1bf7-4d39-8a23-c88787b766d0) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 8192): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/SerializationLowCardinality.cpp:62: DB::SerializationLowCardinality::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x000000001a0107d2
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:42.388152 [ 3810 ] {0671d4b4-5475-4f09-af8b-89e9b21d394c} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:42122) (in query: select v.`Tuple(a UInt32, b UInt32)` from test order by id;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:52:42.411070 [ 3810 ] {0671d4b4-5475-4f09-af8b-89e9b21d394c} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:52:44.656169 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 11 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:44.873534 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:45.566220 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:46.733291 [ 3810 ] {cc3738ee-7ead-4b7e-8d39-742cad16c40e} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=3). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:42894) (in query: optimize table test final;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/SerializationLowCardinality.cpp:62: DB::SerializationLowCardinality::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x000000001a0107d2

2024.11.05 19:52:46.779898 [ 3810 ] {cc3738ee-7ead-4b7e-8d39-742cad16c40e} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/SerializationLowCardinality.cpp:62: DB::SerializationLowCardinality::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x000000001a0107d2

2024.11.05 19:52:52.973293 [ 5795 ] {5a963438-4460-4823-8232-b2608c0f454c} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=20, jobs=13). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:42924) (comment: 00163_column_oriented_formats.sh) (in query: INSERT INTO 00163_column_oriented SELECT ClientEventTime, MobilePhoneModel, ClientIP6 FROM test.hits ORDER BY ClientEventTime, MobilePhoneModel, ClientIP6 LIMIT 100), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:49: DB::threadFunction(DB::CompletedPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d51351d
13. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:90: DB::CompletedPipelineExecutor::execute()::$_0::operator()() const @ 0x000000001d5133aa
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d513355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&>(DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d513339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&>(DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&) @ 0x000000001d51329d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&)::'lambda'()::operator()() @ 0x000000001d5131a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d513115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d5130f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5130d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5130a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:52:53.264426 [ 5795 ] {5a963438-4460-4823-8232-b2608c0f454c} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=20, jobs=13). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:49: DB::threadFunction(DB::CompletedPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d51351d
13. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:90: DB::CompletedPipelineExecutor::execute()::$_0::operator()() const @ 0x000000001d5133aa
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d513355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&>(DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d513339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&>(DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&) @ 0x000000001d51329d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&)::'lambda'()::operator()() @ 0x000000001d5131a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d513115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d5130f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5130d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5130a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:52:53.541255 [ 3810 ] {aea22a95-93b7-42eb-b7db-2aa5cce7a9eb} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:42916) (in query: select v.`LowCardinality(String)` from test order by id;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/SerializationLowCardinality.cpp:62: DB::SerializationLowCardinality::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x000000001a0107d2

2024.11.05 19:52:53.562102 [ 3810 ] {aea22a95-93b7-42eb-b7db-2aa5cce7a9eb} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/SerializationLowCardinality.cpp:62: DB::SerializationLowCardinality::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x000000001a0107d2

2024.11.05 19:52:53.707246 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:55.069111 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:52:57.597553 [ 3810 ] {e55cc9cb-2787-48ef-b00c-2c1b6aa85ebd} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=26, jobs=26). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:53448) (comment: 00163_column_oriented_formats.sh) (in query: SELECT ClientEventTime from 00163_column_oriented), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:52:57.661478 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/tkd/syowywzarcwnaufeyloyjbjrkuubv, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id cc4d645f-ee4c-43f7-87d4-0a2bc1685255, upload has finished false.
2024.11.05 19:52:57.671113 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/tkd/syowywzarcwnaufeyloyjbjrkuubv, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id cc4d645f-ee4c-43f7-87d4-0a2bc1685255, upload has finished false
2024.11.05 19:52:57.716557 [ 3810 ] {e55cc9cb-2787-48ef-b00c-2c1b6aa85ebd} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=26, jobs=26). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:52:57.745117 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/tkd/syowywzarcwnaufeyloyjbjrkuubv, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id cc4d645f-ee4c-43f7-87d4-0a2bc1685255, upload has finished false
2024.11.05 19:52:57.846365 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 34 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:53:02.128255 [ 3810 ] {592a2120-2788-49af-8443-efc3dc6540b6} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=34, jobs=34). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:53488) (comment: 00163_column_oriented_formats.sh) (in query: SELECT MobilePhoneModel from 00163_column_oriented), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:53:02.182873 [ 3810 ] {592a2120-2788-49af-8443-efc3dc6540b6} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=34, jobs=34). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:53:05.105812 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/drw/rvksmtdkbbngqprrasxrrnlchzsgw, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0abd5fc0-0bbf-4e32-84b9-122a42002d90, upload has finished false.
2024.11.05 19:53:05.112254 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/drw/rvksmtdkbbngqprrasxrrnlchzsgw, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0abd5fc0-0bbf-4e32-84b9-122a42002d90, upload has finished false
2024.11.05 19:53:05.159224 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/drw/rvksmtdkbbngqprrasxrrnlchzsgw, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0abd5fc0-0bbf-4e32-84b9-122a42002d90, upload has finished false
2024.11.05 19:53:05.254566 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 16 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:53:06.655812 [ 5793 ] {3d676b52-ef02-42f9-a6b8-88ace938fb91} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=40, jobs=40). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60518) (comment: 00163_column_oriented_formats.sh) (in query: SELECT ClientIP6 from 00163_column_oriented), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:53:06.682261 [ 5793 ] {3d676b52-ef02-42f9-a6b8-88ace938fb91} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=40, jobs=40). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:53:07.108619 [ 3810 ] {ae64af6c-fdb9-498c-8916-4177a7a150b0} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60496) (in query: insert into test select number + 6, number % 2 ? NULL : 'str_' || toString(number) from numbers(3);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:45: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce610d
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375

2024.11.05 19:53:07.128389 [ 3810 ] {ae64af6c-fdb9-498c-8916-4177a7a150b0} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:45: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce610d
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375

2024.11.05 19:53:08.636939 [ 5215 ] {3ac19f1f-8a4f-4aac-aae8-fad45f9b4dc3} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=8, jobs=8). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60504) (comment: 01014_lazy_database_basic.sh) (in query: SELECT * FROM testlazy.slog;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:53:08.664602 [ 5215 ] {3ac19f1f-8a4f-4aac-aae8-fad45f9b4dc3} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=8, jobs=8). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:53:11.738480 [ 5215 ] {b4f91d47-e096-470a-bd0d-62f317aba2ca} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60542) (in query: select v from test order by id;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:53:11.823720 [ 5215 ] {b4f91d47-e096-470a-bd0d-62f317aba2ca} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:53:12.921337 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:53:13.593020 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:53:15.640547 [ 5215 ] {151e2c9e-0909-4d6e-a15b-083b3e17b16f} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:44978) (comment: 00159_parallel_formatting_json_and_friends_2.sh) (in query: SELECT ClientEventTime::DateTime('Asia/Dubai') as a, MobilePhoneModel as b, ClientIP6 as c FROM test.hits ORDER BY a, b, c LIMIT 3000000 Format JSONCompactStringsEachRowWithNames), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:53:16.164828 [ 5215 ] {151e2c9e-0909-4d6e-a15b-083b3e17b16f} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:53:16.811922 [ 5794 ] {f889eded-f1b5-4aec-bcf5-4649280f5ba9} <Error> executeQuery: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::1]:9000 -> [::1]:56018). (NETWORK_ERROR) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:56018) (comment: 02241_filesystem_cache_on_write_operations.sh) (in query: SELECT file_segment_range_begin, file_segment_range_end, size, state FROM ( SELECT file_segment_range_begin, file_segment_range_end, size, state, local_path FROM ( SELECT arrayJoin(cache_paths) AS cache_path, local_path, remote_path FROM system.remote_data_paths ) AS data_paths INNER JOIN system.filesystem_cache AS caches ON data_paths.cache_path = caches.cache_path ) WHERE endsWith(local_path, 'data.bin') FORMAT Vertical), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:169: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e9244
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Server/TCPHandler.cpp:2216: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001d47db72
9. /build/src/Server/TCPHandler.cpp:1055: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b799
10. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:53:16.844896 [ 5794 ] {f889eded-f1b5-4aec-bcf5-4649280f5ba9} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::1]:9000 -> [::1]:56018). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:169: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e9244
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Server/TCPHandler.cpp:2253: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001d487855
9. /build/src/Server/TCPHandler.cpp:2330: DB::TCPHandler::sendLogs() @ 0x000000001d47c5ff
10. /build/src/Server/TCPHandler.cpp:693: DB::TCPHandler::runImpl() @ 0x000000001d471ac2
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:53:16.850208 [ 5794 ] {f889eded-f1b5-4aec-bcf5-4649280f5ba9} <Error> TCPHandler: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::1]:9000 -> [::1]:56018). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:169: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e9244
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Server/TCPHandler.cpp:2216: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001d47db72
9. /build/src/Server/TCPHandler.cpp:1055: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b799
10. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:53:16.859499 [ 5794 ] {f889eded-f1b5-4aec-bcf5-4649280f5ba9} <Warning> TCPHandler: Client has gone away.
2024.11.05 19:53:20.040441 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:53:20.537726 [ 5215 ] {e5e3cc53-cff0-4562-b669-4a5762177939} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:45010) (comment: 00159_parallel_formatting_json_and_friends_2.sh) (in query: SELECT ClientEventTime::DateTime('Asia/Dubai') as a, MobilePhoneModel as b, ClientIP6 as c FROM test.hits ORDER BY a, b, c LIMIT 3000000 Format JSONCompactEachRowWithNamesAndTypes), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:53:20.562934 [ 5215 ] {e5e3cc53-cff0-4562-b669-4a5762177939} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:53:20.875474 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/xof/uurrjnrxqccdabveqxzudjvgmxvba, total size 201326592, count 201326592, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c492533b-de2e-4e64-ac47-3439291fa7a7, upload has finished false.
2024.11.05 19:53:20.876179 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/xof/uurrjnrxqccdabveqxzudjvgmxvba, total size 201326592, count 201326592, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c492533b-de2e-4e64-ac47-3439291fa7a7, upload has finished false
2024.11.05 19:53:20.881712 [ 5793 ] {ff788ea6-b9c9-4e2a-a120-42fcfdda762f} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=29). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:45022) (comment: 01014_lazy_database_basic.sh) (in query: SELECT * FROM testlazy.slog;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:53:20.934852 [ 5793 ] {ff788ea6-b9c9-4e2a-a120-42fcfdda762f} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=29). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:53:20.953606 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/xof/uurrjnrxqccdabveqxzudjvgmxvba, total size 201326592, count 201326592, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c492533b-de2e-4e64-ac47-3439291fa7a7, upload has finished false
2024.11.05 19:53:21.000340 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 44 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:53:24.252086 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=62, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 4 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:53:24.946226 [ 5795 ] {7b2771d3-9a67-4eeb-bdcd-81e49a4fb9be} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=960, jobs=882). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:58862) (in query: select v.`Tuple(a UInt32, b UInt32)` from test order by id;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001d5394f3
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:112: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000001d539234
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000001d5395e9
12. /build/src/Server/TCPHandler.cpp:1023: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b57f
13. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:53:24.958719 [ 5795 ] {7b2771d3-9a67-4eeb-bdcd-81e49a4fb9be} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=960, jobs=882). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001d5394f3
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:112: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000001d539234
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000001d5395e9
12. /build/src/Server/TCPHandler.cpp:1023: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b57f
13. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:53:25.935936 [ 5793 ] {25d383ea-c1c0-49a1-855e-a45b807ab6f4} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:58886) (comment: 00159_parallel_formatting_json_and_friends_2.sh) (in query: SELECT ClientEventTime::DateTime('Asia/Dubai') as a, MobilePhoneModel as b, ClientIP6 as c FROM test.hits ORDER BY a, b, c LIMIT 3000000 Format JSONCompactEachRowWithNamesAndTypes), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:53:26.371817 [ 5793 ] {25d383ea-c1c0-49a1-855e-a45b807ab6f4} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:53:27.418155 [ 5215 ] {13444b15-8ffd-4742-8a05-37fcc66418b6} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:58876) (comment: 00163_column_oriented_formats.sh) (in query: INSERT INTO 00163_column_oriented SELECT ClientEventTime, MobilePhoneModel, ClientIP6 FROM test.hits ORDER BY ClientEventTime, MobilePhoneModel, ClientIP6 LIMIT 100), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:53:28.380573 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column Referer): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_3_3_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 26 with max_rows_to_read = 6608): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:53:30.260422 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:53:31.288991 [ 5215 ] {13444b15-8ffd-4742-8a05-37fcc66418b6} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:53:33.300637 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:53:35.484126 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/rzv/kpmdacfsklwnrnixqjaeazlwbhkzl, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 23c0f2a4-f735-4704-890a-10f01dd5b218, upload has finished false.
2024.11.05 19:53:35.489326 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/rzv/kpmdacfsklwnrnixqjaeazlwbhkzl, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 23c0f2a4-f735-4704-890a-10f01dd5b218, upload has finished false
2024.11.05 19:53:35.515420 [ 3610 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Removing temporary directory /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:53:35.516453 [ 3610 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:53:35.563723 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/rzv/kpmdacfsklwnrnixqjaeazlwbhkzl, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 23c0f2a4-f735-4704-890a-10f01dd5b218, upload has finished false
2024.11.05 19:53:35.690526 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 27 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:53:37.087180 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:53:38.686668 [ 5795 ] {21cb64b1-408a-4bf2-b5f7-a2b58250e1cc} <Error> executeQuery: Code: 475. DB::Exception: Multiple WITH FILL for identical expressions is not supported in ORDER BY. (INVALID_WITH_FILL_EXPRESSION) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:35708) (comment: 02018_multiple_with_fill_for_the_same_column.sql) (in query: SELECT x, y FROM (SELECT 5 AS x, 'Hello' AS y) ORDER BY x WITH FILL FROM 3 TO 7, y, x WITH FILL FROM 1 TO 10;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Processors/Transforms/FillingTransform.cpp:251: DB::FillingTransform::FillingTransform(DB::Block const&, DB::SortDescription const&, DB::SortDescription const&, std::shared_ptr<DB::InterpolateDescription>, bool) @ 0x000000001dcea67c
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::FillingTransform* std::construct_at[abi:v15000]<DB::FillingTransform, DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>, bool const&, DB::FillingTransform*>(DB::FillingTransform*, DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>&&, bool const&) @ 0x000000001dce91d8
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::FillingTransform>>::construct[abi:v15000]<DB::FillingTransform, DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>, bool const&, void, void>(std::allocator<DB::FillingTransform>&, DB::FillingTransform*, DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>&&, bool const&) @ 0x000000001dce8f41
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::FillingTransform, std::allocator<DB::FillingTransform>>::__shared_ptr_emplace[abi:v15000]<DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>, bool const&>(std::allocator<DB::FillingTransform>, DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>&&, bool const&) @ 0x000000001dce8c2d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::FillingTransform> std::allocate_shared[abi:v15000]<DB::FillingTransform, std::allocator<DB::FillingTransform>, DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>, bool const&, void>(std::allocator<DB::FillingTransform> const&, DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>&&, bool const&) @ 0x000000001dce8a7a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::FillingTransform> std::make_shared[abi:v15000]<DB::FillingTransform, DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>, bool const&, void>(DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>&&, bool const&) @ 0x000000001dce7e66
12. /build/src/Processors/QueryPlan/FillingStep.cpp:53: DB::FillingStep::transformPipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&)::$_0::operator()(DB::Block const&, DB::Pipe::StreamType) const @ 0x000000001dce7bd1
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001dce7b27
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001dce7ae5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001dce7aa5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001dce7a4c
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019e52128
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019e520a4
19. /build/src/QueryPipeline/Pipe.cpp:593: DB::Pipe::addSimpleTransform(std::function<std::shared_ptr<DB::IProcessor> (DB::Block const&, DB::Pipe::StreamType)> const&)::$_0::operator()(DB::OutputPort*&, DB::Pipe::StreamType) const @ 0x0000000019e4062e
20. /build/src/QueryPipeline/Pipe.cpp:633: DB::Pipe::addSimpleTransform(std::function<std::shared_ptr<DB::IProcessor> (DB::Block const&, DB::Pipe::StreamType)> const&) @ 0x0000000019e40515
21. /build/src/QueryPipeline/QueryPipelineBuilder.cpp:136: DB::QueryPipelineBuilder::addSimpleTransform(std::function<std::shared_ptr<DB::IProcessor> (DB::Block const&, DB::Pipe::StreamType)> const&) @ 0x0000000019e74a4e
22. /build/src/Processors/QueryPlan/FillingStep.cpp:48: DB::FillingStep::transformPipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dce74e2
23. /build/src/Processors/QueryPlan/ITransformingStep.cpp:41: DB::ITransformingStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd099a6
24. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
25. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
26. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
27. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
28. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
29. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
30. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
31. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619

2024.11.05 19:53:38.701931 [ 5795 ] {21cb64b1-408a-4bf2-b5f7-a2b58250e1cc} <Error> TCPHandler: Code: 475. DB::Exception: Multiple WITH FILL for identical expressions is not supported in ORDER BY. (INVALID_WITH_FILL_EXPRESSION), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Processors/Transforms/FillingTransform.cpp:251: DB::FillingTransform::FillingTransform(DB::Block const&, DB::SortDescription const&, DB::SortDescription const&, std::shared_ptr<DB::InterpolateDescription>, bool) @ 0x000000001dcea67c
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::FillingTransform* std::construct_at[abi:v15000]<DB::FillingTransform, DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>, bool const&, DB::FillingTransform*>(DB::FillingTransform*, DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>&&, bool const&) @ 0x000000001dce91d8
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::FillingTransform>>::construct[abi:v15000]<DB::FillingTransform, DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>, bool const&, void, void>(std::allocator<DB::FillingTransform>&, DB::FillingTransform*, DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>&&, bool const&) @ 0x000000001dce8f41
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::FillingTransform, std::allocator<DB::FillingTransform>>::__shared_ptr_emplace[abi:v15000]<DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>, bool const&>(std::allocator<DB::FillingTransform>, DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>&&, bool const&) @ 0x000000001dce8c2d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::FillingTransform> std::allocate_shared[abi:v15000]<DB::FillingTransform, std::allocator<DB::FillingTransform>, DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>, bool const&, void>(std::allocator<DB::FillingTransform> const&, DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>&&, bool const&) @ 0x000000001dce8a7a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::FillingTransform> std::make_shared[abi:v15000]<DB::FillingTransform, DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>, bool const&, void>(DB::Block const&, DB::SortDescription&, DB::SortDescription&, std::shared_ptr<DB::InterpolateDescription>&&, bool const&) @ 0x000000001dce7e66
12. /build/src/Processors/QueryPlan/FillingStep.cpp:53: DB::FillingStep::transformPipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&)::$_0::operator()(DB::Block const&, DB::Pipe::StreamType) const @ 0x000000001dce7bd1
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001dce7b27
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001dce7ae5
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001dce7aa5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001dce7a4c
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019e52128
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019e520a4
19. /build/src/QueryPipeline/Pipe.cpp:593: DB::Pipe::addSimpleTransform(std::function<std::shared_ptr<DB::IProcessor> (DB::Block const&, DB::Pipe::StreamType)> const&)::$_0::operator()(DB::OutputPort*&, DB::Pipe::StreamType) const @ 0x0000000019e4062e
20. /build/src/QueryPipeline/Pipe.cpp:633: DB::Pipe::addSimpleTransform(std::function<std::shared_ptr<DB::IProcessor> (DB::Block const&, DB::Pipe::StreamType)> const&) @ 0x0000000019e40515
21. /build/src/QueryPipeline/QueryPipelineBuilder.cpp:136: DB::QueryPipelineBuilder::addSimpleTransform(std::function<std::shared_ptr<DB::IProcessor> (DB::Block const&, DB::Pipe::StreamType)> const&) @ 0x0000000019e74a4e
22. /build/src/Processors/QueryPlan/FillingStep.cpp:48: DB::FillingStep::transformPipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dce74e2
23. /build/src/Processors/QueryPlan/ITransformingStep.cpp:41: DB::ITransformingStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd099a6
24. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
25. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
26. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
27. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
28. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
29. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
30. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
31. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619

2024.11.05 19:53:41.904101 [ 5795 ] {636d7b6d-5a2a-4be8-b290-7b802c971c58} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:35718) (comment: 00163_column_oriented_formats.sh) (in query: SELECT MobilePhoneModel from 00163_column_oriented), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:53:41.958132 [ 5795 ] {636d7b6d-5a2a-4be8-b290-7b802c971c58} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=5, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:53:45.059261 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/utg/cjyebjxpufwfjuuiakbkwulazvyqb, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 7601184c-da1d-483b-a902-0d340e9ba2a3, upload has finished false.
2024.11.05 19:53:45.068649 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/utg/cjyebjxpufwfjuuiakbkwulazvyqb, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 7601184c-da1d-483b-a902-0d340e9ba2a3, upload has finished false
2024.11.05 19:53:45.119420 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/utg/cjyebjxpufwfjuuiakbkwulazvyqb, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 7601184c-da1d-483b-a902-0d340e9ba2a3, upload has finished false
2024.11.05 19:53:45.245220 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 20 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:53:46.330181 [ 5795 ] {560135ac-771b-4659-815d-6a36a8c816b2} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=21, jobs=21). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:36720) (comment: 00163_column_oriented_formats.sh) (in query: SELECT ClientIP6 from 00163_column_oriented), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:53:46.450545 [ 5795 ] {560135ac-771b-4659-815d-6a36a8c816b2} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=21, jobs=21). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:53:49.564051 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 8 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:53:53.251322 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=1): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 4 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:02.085185 [ 3641 ] {} <Error> void DB::SystemLog<DB::PartLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::PartLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:02.389309 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:03.769452 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/nse/axfzxfbuymknzgodqudmskleqmypq, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ec282527-0bd7-453c-b5e2-415c4932b280, upload has finished false.
2024.11.05 19:54:03.778427 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/nse/axfzxfbuymknzgodqudmskleqmypq, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ec282527-0bd7-453c-b5e2-415c4932b280, upload has finished false
2024.11.05 19:54:03.842508 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/nse/axfzxfbuymknzgodqudmskleqmypq, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ec282527-0bd7-453c-b5e2-415c4932b280, upload has finished false
2024.11.05 19:54:03.877344 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 26 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:06.548767 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:07.045478 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:08.414529 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:10.928248 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:12.195579 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0a233324-63e4-4351-8154-d23ef4927e18::202411_1_96_19}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:12.798278 [ 5215 ] {4f72c5f8-ed5a-4409-a01f-61e36e9fa8a5} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:51540) (comment: 02125_lz4_compression_bug_JSONCompactEachRow.sh) (in query: INSERT INTO file SELECT * FROM numbers(10000)), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.h:298: DB::ParallelFormattingOutputFormat::scheduleFormatterThreadForUnitWithNumber(unsigned long, unsigned long) @ 0x000000001d7bfaf0
10. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.cpp:94: DB::ParallelFormattingOutputFormat::addChunk(DB::Chunk, DB::ParallelFormattingOutputFormat::ProcessingUnitType, bool) @ 0x000000001d7be7e8
11. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.h:121: DB::ParallelFormattingOutputFormat::writePrefix() @ 0x000000001d7c03c4
12. /build/src/Processors/Formats/IOutputFormat.h:103: DB::IOutputFormat::writePrefixIfNeeded() @ 0x000000000d8378d3
13. /build/src/Processors/Formats/IOutputFormat.cpp:116: DB::IOutputFormat::write(DB::Block const&) @ 0x000000001d5483e0
14. /build/src/Storages/StorageFile.cpp:1778: DB::StorageFileSink::consume(DB::Chunk) @ 0x000000001c3916ab
15. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
16. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:114: DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x000000001da41c5a
24. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work() @ 0x000000001da4169e
25. /build/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001d5343e3
26. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001d534120
27. /build/src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d5166a1
28. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
29. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915

2024.11.05 19:54:12.888819 [ 5215 ] {4f72c5f8-ed5a-4409-a01f-61e36e9fa8a5} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.h:298: DB::ParallelFormattingOutputFormat::scheduleFormatterThreadForUnitWithNumber(unsigned long, unsigned long) @ 0x000000001d7bfaf0
10. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.cpp:94: DB::ParallelFormattingOutputFormat::addChunk(DB::Chunk, DB::ParallelFormattingOutputFormat::ProcessingUnitType, bool) @ 0x000000001d7be7e8
11. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.h:121: DB::ParallelFormattingOutputFormat::writePrefix() @ 0x000000001d7c03c4
12. /build/src/Processors/Formats/IOutputFormat.h:103: DB::IOutputFormat::writePrefixIfNeeded() @ 0x000000000d8378d3
13. /build/src/Processors/Formats/IOutputFormat.cpp:116: DB::IOutputFormat::write(DB::Block const&) @ 0x000000001d5483e0
14. /build/src/Storages/StorageFile.cpp:1778: DB::StorageFileSink::consume(DB::Chunk) @ 0x000000001c3916ab
15. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
16. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:114: DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x000000001da41c5a
24. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work() @ 0x000000001da4169e
25. /build/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001d5343e3
26. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001d534120
27. /build/src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d5166a1
28. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
29. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915

2024.11.05 19:54:20.012084 [ 5794 ] {5be48d5b-3675-4b12-a817-4bb903ba9df9} <Error> executeQuery: Code: 36. DB::Exception: Cannot load time zone /abc: In scope SELECT toDateTime(0, '/abc'). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:51552) (comment: 02505_forbid_paths_in_datetime_timezone.sql) (in query: select toDateTime(0, '/abc');), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:54:20.030339 [ 5794 ] {5be48d5b-3675-4b12-a817-4bb903ba9df9} <Error> TCPHandler: Code: 36. DB::Exception: Cannot load time zone /abc: In scope SELECT toDateTime(0, '/abc'). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:54:20.276891 [ 5794 ] {70c630ee-820b-4208-89b4-071d1d462cb3} <Error> executeQuery: Code: 36. DB::Exception: Cannot load time zone ./abc: In scope SELECT toDateTime(0, './abc'). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:51552) (comment: 02505_forbid_paths_in_datetime_timezone.sql) (in query: select toDateTime(0, './abc');), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:54:20.284282 [ 5794 ] {70c630ee-820b-4208-89b4-071d1d462cb3} <Error> TCPHandler: Code: 36. DB::Exception: Cannot load time zone ./abc: In scope SELECT toDateTime(0, './abc'). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:54:20.405354 [ 5794 ] {9ade6484-a0a2-48d8-b65a-dd9b845bd0ba} <Error> executeQuery: Code: 36. DB::Exception: Cannot load time zone ../abc: In scope SELECT toDateTime(0, '../abc'). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:51552) (comment: 02505_forbid_paths_in_datetime_timezone.sql) (in query: select toDateTime(0, '../abc');), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:54:20.426151 [ 5794 ] {9ade6484-a0a2-48d8-b65a-dd9b845bd0ba} <Error> TCPHandler: Code: 36. DB::Exception: Cannot load time zone ../abc: In scope SELECT toDateTime(0, '../abc'). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:54:20.587295 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/jht/mwckjjrznzyrvetgahxthlwmeokib, total size 167772160, count 167772160, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id cbf83621-6dd0-4d24-8c94-59f487ace441, upload has finished false.
2024.11.05 19:54:20.591174 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/jht/mwckjjrznzyrvetgahxthlwmeokib, total size 167772160, count 167772160, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id cbf83621-6dd0-4d24-8c94-59f487ace441, upload has finished false
2024.11.05 19:54:20.656320 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/jht/mwckjjrznzyrvetgahxthlwmeokib, total size 167772160, count 167772160, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id cbf83621-6dd0-4d24-8c94-59f487ace441, upload has finished false
2024.11.05 19:54:20.703157 [ 5794 ] {5cecaa3e-252e-4f21-9272-e0f53c2d1b86} <Error> executeQuery: Code: 36. DB::Exception: Cannot load time zone ~/abc: In scope SELECT toDateTime(0, '~/abc'). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:51552) (comment: 02505_forbid_paths_in_datetime_timezone.sql) (in query: select toDateTime(0, '~/abc');), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:54:20.707012 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 38 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:20.717804 [ 5794 ] {5cecaa3e-252e-4f21-9272-e0f53c2d1b86} <Error> TCPHandler: Code: 36. DB::Exception: Cannot load time zone ~/abc: In scope SELECT toDateTime(0, '~/abc'). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:54:20.887253 [ 5794 ] {568d1ff1-6fd9-48b7-b900-1fd4f20a4651} <Error> executeQuery: Code: 36. DB::Exception: Cannot load time zone abc/../../cba: In scope SELECT toDateTime(0, 'abc/../../cba'). (BAD_ARGUMENTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:51552) (comment: 02505_forbid_paths_in_datetime_timezone.sql) (in query: select toDateTime(0, 'abc/../../cba');), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:54:20.900128 [ 5794 ] {568d1ff1-6fd9-48b7-b900-1fd4f20a4651} <Error> TCPHandler: Code: 36. DB::Exception: Cannot load time zone abc/../../cba: In scope SELECT toDateTime(0, 'abc/../../cba'). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Common/DateLUTImpl.cpp:77: DateLUTImpl::DateLUTImpl(String const&) @ 0x00000000132a1994
7. /build/src/Common/DateLUT.cpp:157: DateLUT::getImplementation(String const&) const @ 0x000000001329c22a
8. /build/src/Common/DateLUT.h:62: TimezoneMixin::TimezoneMixin(String const&) @ 0x0000000019ecf469
9. /build/src/DataTypes/DataTypeDateTime.cpp:11: DB::DataTypeDateTime::DataTypeDateTime(String const&) @ 0x0000000019eced4e
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::DataTypeDateTime* std::construct_at[abi:v15000]<DB::DataTypeDateTime, String, DB::DataTypeDateTime*>(DB::DataTypeDateTime*, String&&) @ 0x000000000af70366
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::DataTypeDateTime>>::construct[abi:v15000]<DB::DataTypeDateTime, String, void, void>(std::allocator<DB::DataTypeDateTime>&, DB::DataTypeDateTime*, String&&) @ 0x000000000af702e1
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>>::__shared_ptr_emplace[abi:v15000]<String>(std::allocator<DB::DataTypeDateTime>, String&&) @ 0x000000000af70267
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::DataTypeDateTime> std::allocate_shared[abi:v15000]<DB::DataTypeDateTime, std::allocator<DB::DataTypeDateTime>, String, void>(std::allocator<DB::DataTypeDateTime> const&, String&&) @ 0x000000000af70197
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::DataTypeDateTime> std::make_shared[abi:v15000]<DB::DataTypeDateTime, String, void>(String&&) @ 0x000000000af6f0e6
15. /build/src/Functions/FunctionsConversion.cpp:2078: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImplRemovedNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acdcd15
16. /build/src/Functions/FunctionsConversion.cpp:1999: auto DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const::'lambda'(auto const&)::operator()<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>>(auto const&) const @ 0x000000000acdc940
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000000acdc8fd
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000000acdc8bd
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000000acdc87d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000000acdc845
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000000af6ebde
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000000af6d95d
23. /build/src/Functions/FunctionsConversion.cpp:1994: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeDefaultImplementationForNulls(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::function<std::shared_ptr<DB::IDataType const> (std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&)> const&) @ 0x000000000acdc67b
24. /build/src/Functions/FunctionsConversion.cpp:2000: DB::(anonymous namespace)::FunctionConvert<DB::DataTypeDateTime, DB::(anonymous namespace)::NameToDateTime, DB::(anonymous namespace)::ToDateTimeMonotonicity>::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000acb113a
25. /build/src/Functions/IFunctionAdaptors.h:136: DB::FunctionToOverloadResolverAdaptor::getReturnTypeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x000000000af6a350
26. /build/src/Functions/IFunction.cpp:522: DB::IFunctionOverloadResolver::getReturnTypeWithoutLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065be7
27. /build/src/Functions/IFunction.cpp:468: DB::IFunctionOverloadResolver::getReturnType(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065636
28. /build/src/Functions/IFunction.cpp:483: DB::IFunctionOverloadResolver::build(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) const @ 0x0000000019065c41
29. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6160: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001b0be110
30. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6487: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a2cea
31. /build/src/Analyzer/Passes/QueryAnalysisPass.cpp:6640: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001b0a1221

2024.11.05 19:54:23.653264 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:25.450228 [ 5215 ] {fdf910b8-3e67-423e-9908-5dd365b2a837} <Error> executeQuery: Code: 44. DB::Exception: Columns UInt32 and DateTime64 cannot be compared. (ILLEGAL_COLUMN) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52282) (comment: 02010_array_index_bad_cast.sql) (in query: SELECT has(materialize(CAST(['2021-07-14'] AS Array(LowCardinality(Nullable(DateTime))))), materialize('2021-07-14'::DateTime64(7)));), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000a8f2fcc
6. /build/src/Functions/array/arrayIndex.h:124: void DB::Impl::Main<DB::HasAction, true, unsigned long, unsigned long>::process<3ul, DB::IColumn, DB::IColumn>(DB::IColumn const&, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::IColumn const&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul>&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const*, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const*) @ 0x00000000120fc874
7. /build/src/Functions/array/arrayIndex.h:188: void DB::Impl::Main<DB::HasAction, true, unsigned long, unsigned long>::vector<DB::IColumn, DB::IColumn>(DB::IColumn const&, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::IColumn const&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul>&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const*, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const*) @ 0x0000000012096966
8. /build/src/Functions/array/arrayIndex.h:857: DB::FunctionArrayIndex<DB::HasAction, DB::NameHas>::dispatchConvertedLowCardinalityColumns(DB::FunctionArrayIndex<DB::HasAction, DB::NameHas>::ExecutionData&) @ 0x0000000012d52daa
9. /build/src/Functions/array/arrayIndex.h:813: DB::FunctionArrayIndex<DB::HasAction, DB::NameHas>::executeLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) @ 0x0000000012d5129c
10. /build/src/Functions/array/arrayIndex.h:585: DB::FunctionArrayIndex<DB::HasAction, DB::NameHas>::executeOnNonNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&) const @ 0x0000000012d504be
11. /build/src/Functions/array/arrayIndex.h:516: DB::FunctionArrayIndex<DB::HasAction, DB::NameHas>::executeArrayImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&) const @ 0x0000000012d4ff61
12. /build/src/Functions/array/arrayIndex.h:0: DB::FunctionArrayIndex<DB::HasAction, DB::NameHas>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x0000000012d4f880
13. /build/src/Functions/IFunction.h:443: DB::IFunction::executeImplDryRun(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000a8efa99
14. /build/src/Functions/IFunctionAdaptors.h:26: DB::FunctionToExecutableFunctionAdaptor::executeDryRunImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000af6b1ad
15. /build/src/Functions/IFunction.cpp:277: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062f06
16. /build/src/Functions/IFunction.cpp:340: DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000001906408e
17. /build/src/Functions/IFunction.cpp:411: DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000190650b5
18. /build/src/Interpreters/ActionsDAG.cpp:662: DB::executeActionForPartialResult(DB::ActionsDAG::Node const*, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>, unsigned long) @ 0x000000001a43c967
19. /build/src/Interpreters/ActionsDAG.cpp:829: DB::ActionsDAG::evaluatePartialResult(std::unordered_map<DB::ActionsDAG::Node const*, DB::ColumnWithTypeAndName, std::hash<DB::ActionsDAG::Node const*>, std::equal_to<DB::ActionsDAG::Node const*>, std::allocator<std::pair<DB::ActionsDAG::Node const* const, DB::ColumnWithTypeAndName>>>&, std::vector<DB::ActionsDAG::Node const*, std::allocator<DB::ActionsDAG::Node const*>> const&, unsigned long, bool) @ 0x000000001a43c59e
20. /build/src/Interpreters/ActionsDAG.cpp:739: DB::ActionsDAG::updateHeader(DB::Block) const @ 0x000000001a43ba80
21. /build/src/Processors/Transforms/ExpressionTransform.cpp:8: DB::ExpressionTransform::transformHeader(DB::Block, DB::ActionsDAG const&) @ 0x000000001da429d1
22. /build/src/Processors/QueryPlan/ExpressionStep.cpp:31: DB::ExpressionStep::ExpressionStep(DB::DataStream const&, std::shared_ptr<DB::ActionsDAG> const&) @ 0x000000001dce5926
23. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::ExpressionStep>::__unique_single std::make_unique[abi:v15000]<DB::ExpressionStep, DB::DataStream const&, std::shared_ptr<DB::ActionsDAG> const&>(DB::DataStream const&, std::shared_ptr<DB::ActionsDAG> const&) @ 0x000000001b467fd1
24. /build/src/Planner/Planner.cpp:336: DB::(anonymous namespace)::addExpressionStep(DB::QueryPlan&, std::shared_ptr<DB::ActionsDAG> const&, String const&, std::vector<std::shared_ptr<DB::ActionsDAG>, std::allocator<std::shared_ptr<DB::ActionsDAG>>>&) @ 0x000000001b513b1a
25. /build/src/Planner/Planner.cpp:1567: DB::Planner::buildPlanForQueryNode() @ 0x000000001b51108d
26. /build/src/Planner/Planner.cpp:0: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000001b50be55
27. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:234: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000001b50905d
28. /build/src/Interpreters/executeQuery.cpp:1146: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2812
29. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
30. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
31. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5

2024.11.05 19:54:25.473659 [ 5215 ] {fdf910b8-3e67-423e-9908-5dd365b2a837} <Error> TCPHandler: Code: 44. DB::Exception: Columns UInt32 and DateTime64 cannot be compared. (ILLEGAL_COLUMN), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000a8f2fcc
6. /build/src/Functions/array/arrayIndex.h:124: void DB::Impl::Main<DB::HasAction, true, unsigned long, unsigned long>::process<3ul, DB::IColumn, DB::IColumn>(DB::IColumn const&, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::IColumn const&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul>&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const*, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const*) @ 0x00000000120fc874
7. /build/src/Functions/array/arrayIndex.h:188: void DB::Impl::Main<DB::HasAction, true, unsigned long, unsigned long>::vector<DB::IColumn, DB::IColumn>(DB::IColumn const&, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::IColumn const&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul>&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const*, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const*) @ 0x0000000012096966
8. /build/src/Functions/array/arrayIndex.h:857: DB::FunctionArrayIndex<DB::HasAction, DB::NameHas>::dispatchConvertedLowCardinalityColumns(DB::FunctionArrayIndex<DB::HasAction, DB::NameHas>::ExecutionData&) @ 0x0000000012d52daa
9. /build/src/Functions/array/arrayIndex.h:813: DB::FunctionArrayIndex<DB::HasAction, DB::NameHas>::executeLowCardinality(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&) @ 0x0000000012d5129c
10. /build/src/Functions/array/arrayIndex.h:585: DB::FunctionArrayIndex<DB::HasAction, DB::NameHas>::executeOnNonNullable(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&) const @ 0x0000000012d504be
11. /build/src/Functions/array/arrayIndex.h:516: DB::FunctionArrayIndex<DB::HasAction, DB::NameHas>::executeArrayImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&) const @ 0x0000000012d4ff61
12. /build/src/Functions/array/arrayIndex.h:0: DB::FunctionArrayIndex<DB::HasAction, DB::NameHas>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x0000000012d4f880
13. /build/src/Functions/IFunction.h:443: DB::IFunction::executeImplDryRun(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000a8efa99
14. /build/src/Functions/IFunctionAdaptors.h:26: DB::FunctionToExecutableFunctionAdaptor::executeDryRunImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000af6b1ad
15. /build/src/Functions/IFunction.cpp:277: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000019062f06
16. /build/src/Functions/IFunction.cpp:340: DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000001906408e
17. /build/src/Functions/IFunction.cpp:411: DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x00000000190650b5
18. /build/src/Interpreters/ActionsDAG.cpp:662: DB::executeActionForPartialResult(DB::ActionsDAG::Node const*, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>, unsigned long) @ 0x000000001a43c967
19. /build/src/Interpreters/ActionsDAG.cpp:829: DB::ActionsDAG::evaluatePartialResult(std::unordered_map<DB::ActionsDAG::Node const*, DB::ColumnWithTypeAndName, std::hash<DB::ActionsDAG::Node const*>, std::equal_to<DB::ActionsDAG::Node const*>, std::allocator<std::pair<DB::ActionsDAG::Node const* const, DB::ColumnWithTypeAndName>>>&, std::vector<DB::ActionsDAG::Node const*, std::allocator<DB::ActionsDAG::Node const*>> const&, unsigned long, bool) @ 0x000000001a43c59e
20. /build/src/Interpreters/ActionsDAG.cpp:739: DB::ActionsDAG::updateHeader(DB::Block) const @ 0x000000001a43ba80
21. /build/src/Processors/Transforms/ExpressionTransform.cpp:8: DB::ExpressionTransform::transformHeader(DB::Block, DB::ActionsDAG const&) @ 0x000000001da429d1
22. /build/src/Processors/QueryPlan/ExpressionStep.cpp:31: DB::ExpressionStep::ExpressionStep(DB::DataStream const&, std::shared_ptr<DB::ActionsDAG> const&) @ 0x000000001dce5926
23. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::ExpressionStep>::__unique_single std::make_unique[abi:v15000]<DB::ExpressionStep, DB::DataStream const&, std::shared_ptr<DB::ActionsDAG> const&>(DB::DataStream const&, std::shared_ptr<DB::ActionsDAG> const&) @ 0x000000001b467fd1
24. /build/src/Planner/Planner.cpp:336: DB::(anonymous namespace)::addExpressionStep(DB::QueryPlan&, std::shared_ptr<DB::ActionsDAG> const&, String const&, std::vector<std::shared_ptr<DB::ActionsDAG>, std::allocator<std::shared_ptr<DB::ActionsDAG>>>&) @ 0x000000001b513b1a
25. /build/src/Planner/Planner.cpp:1567: DB::Planner::buildPlanForQueryNode() @ 0x000000001b51108d
26. /build/src/Planner/Planner.cpp:0: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000001b50be55
27. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:234: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000001b50905d
28. /build/src/Interpreters/executeQuery.cpp:1146: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2812
29. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
30. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
31. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5

2024.11.05 19:54:25.923927 [ 3609 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Removing temporary directory /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:54:25.924682 [ 3609 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 19:54:26.222318 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 10 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:30.658465 [ 5215 ] {912b7f80-a904-41db-85e2-02ff494625e0} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52310) (comment: 01712_no_adaptive_granularity_vertical_merge.sql) (in query: INSERT INTO old_school_table VALUES ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6

2024.11.05 19:54:30.676992 [ 5215 ] {912b7f80-a904-41db-85e2-02ff494625e0} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6

2024.11.05 19:54:31.063871 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:34.748331 [ 5795 ] {f5a57887-574c-43db-90b9-2e733a1379b5} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=11): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:51026) (comment: 00063_loyalty_joins.sql) (in query: SELECT loyalty, count() FROM ( SELECT UserID FROM test.hits ) ANY LEFT JOIN ( SELECT UserID, sum(SearchEngineID = 2) AS yandex, sum(SearchEngineID = 3) AS google, toInt8(if(yandex > google, yandex / (yandex + google), -google / (yandex + google)) * 10) AS loyalty FROM test.hits WHERE (SearchEngineID = 2) OR (SearchEngineID = 3) GROUP BY UserID HAVING (yandex + google) > 10 ) USING UserID GROUP BY loyalty ORDER BY loyalty ASC;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:54:34.939121 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {d830f0a7-c4af-4441-961e-a7e3a729e583::202411_1_116_23}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:38.478411 [ 5795 ] {f5a57887-574c-43db-90b9-2e733a1379b5} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=11): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:54:38.570397 [ 5794 ] {ebb82820-4114-4453-b850-fd499e101aef} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46010) (comment: 02881_system_detached_parts_modification_time.gen.sql) (in query: select 'before detach', now()-modification_time < 10 from system.detached_parts where database = currentDatabase() and table = 'data_wide';), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:54:38.741980 [ 5794 ] {ebb82820-4114-4453-b850-fd499e101aef} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:54:38.826074 [ 3609 ] {} <Warning> system.trace_log (4ff3d7e3-1bf7-4d39-8a23-c88787b766d0): Removing temporary directory /var/lib/clickhouse/disks/s3/store/4ff/4ff3d7e3-1bf7-4d39-8a23-c88787b766d0/tmp_merge_202411_1_106_20/
2024.11.05 19:54:38.826610 [ 3609 ] {} <Warning> system.trace_log (4ff3d7e3-1bf7-4d39-8a23-c88787b766d0): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3/store/4ff/4ff3d7e3-1bf7-4d39-8a23-c88787b766d0/tmp_merge_202411_1_106_20/
2024.11.05 19:54:38.880273 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_106_20}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=6). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:176: DB::WriteBufferFromS3::preFinalize() @ 0x00000000199098b4
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:43: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce60f4
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
24. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
25. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
26. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
27. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
28. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:41.468293 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:41.943784 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:42.273250 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:43.386628 [ 5215 ] {c739564c-901c-45f1-b620-3a837de588ec} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:38336) (comment: 02125_lz4_compression_bug_JSONCompactEachRow.sh) (in query: INSERT INTO file SELECT * FROM numbers(1000000)), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.h:298: DB::ParallelFormattingOutputFormat::scheduleFormatterThreadForUnitWithNumber(unsigned long, unsigned long) @ 0x000000001d7bfaf0
10. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.cpp:94: DB::ParallelFormattingOutputFormat::addChunk(DB::Chunk, DB::ParallelFormattingOutputFormat::ProcessingUnitType, bool) @ 0x000000001d7be7e8
11. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.h:159: DB::ParallelFormattingOutputFormat::consume(DB::Chunk) @ 0x000000001d7c0179
12. /build/src/Processors/Formats/IOutputFormat.cpp:116: DB::IOutputFormat::write(DB::Block const&) @ 0x000000001d548429
13. /build/src/Storages/StorageFile.cpp:1778: DB::StorageFileSink::consume(DB::Chunk) @ 0x000000001c3916ab
14. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
15. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
22. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:114: DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x000000001da41c5a
23. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work() @ 0x000000001da4169e
24. /build/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001d5343e3
25. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001d534120
26. /build/src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d5166a1
27. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
28. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5

2024.11.05 19:54:43.389785 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_106_20}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column event): (while reading from part /var/lib/clickhouse/disks/s3/store/4ff/4ff3d7e3-1bf7-4d39-8a23-c88787b766d0/202411_1_100_19/ in table system.trace_log (4ff3d7e3-1bf7-4d39-8a23-c88787b766d0) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 8192): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/SerializationLowCardinality.cpp:62: DB::SerializationLowCardinality::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x000000001a0107d2
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:43.463523 [ 5215 ] {c739564c-901c-45f1-b620-3a837de588ec} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=1, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.h:298: DB::ParallelFormattingOutputFormat::scheduleFormatterThreadForUnitWithNumber(unsigned long, unsigned long) @ 0x000000001d7bfaf0
10. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.cpp:94: DB::ParallelFormattingOutputFormat::addChunk(DB::Chunk, DB::ParallelFormattingOutputFormat::ProcessingUnitType, bool) @ 0x000000001d7be7e8
11. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.h:159: DB::ParallelFormattingOutputFormat::consume(DB::Chunk) @ 0x000000001d7c0179
12. /build/src/Processors/Formats/IOutputFormat.cpp:116: DB::IOutputFormat::write(DB::Block const&) @ 0x000000001d548429
13. /build/src/Storages/StorageFile.cpp:1778: DB::StorageFileSink::consume(DB::Chunk) @ 0x000000001c3916ab
14. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
15. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
22. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:114: DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x000000001da41c5a
23. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work() @ 0x000000001da4169e
24. /build/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001d5343e3
25. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001d534120
26. /build/src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d5166a1
27. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
28. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5

2024.11.05 19:54:43.698057 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/fzx/keimxfxbppqdpmaxnxpaydqxsvxeg, total size 234881024, count 234881024, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c8307d08-4561-423c-b177-dc22b6a67c68, upload has finished false.
2024.11.05 19:54:43.709634 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/fzx/keimxfxbppqdpmaxnxpaydqxsvxeg, total size 234881024, count 234881024, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c8307d08-4561-423c-b177-dc22b6a67c68, upload has finished false
2024.11.05 19:54:43.795626 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/fzx/keimxfxbppqdpmaxnxpaydqxsvxeg, total size 234881024, count 234881024, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c8307d08-4561-423c-b177-dc22b6a67c68, upload has finished false
2024.11.05 19:54:44.141456 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 50 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:44.757011 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:45.008432 [ 5215 ] {34a75735-e57a-4615-b1ad-f9f677e0dd66} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:38358) (comment: 00141_transform.sql) (in query: SELECT transform(SearchEngineID, [2, 3], ['Яндекс', 'Google'], 'Остальные') AS title, count() AS c FROM test.hits WHERE SearchEngineID != 0 GROUP BY title HAVING c > 0 ORDER BY c DESC LIMIT 10;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:54:45.025129 [ 5215 ] {34a75735-e57a-4615-b1ad-f9f677e0dd66} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:54:49.454773 [ 3800 ] {3c8f67d0-6141-4697-a7f8-814cd3a3dd46} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=29). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:38374) (comment: 02125_lz4_compression_bug_JSONCompactEachRow.sh) (in query: SELECT max(x) FROM file), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:54:49.706478 [ 3800 ] {3c8f67d0-6141-4697-a7f8-814cd3a3dd46} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=29, jobs=29). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:54:50.848102 [ 5794 ] {bae2e644-fa9f-4475-ac4c-fbfb877930a3} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:38380) (comment: 02863_mutation_where_in_set_result_cache_pipeline_stuck_bug.sql) (in query: insert into tab select number, number from numbers(20);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d

2024.11.05 19:54:50.856742 [ 5794 ] {bae2e644-fa9f-4475-ac4c-fbfb877930a3} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d

2024.11.05 19:54:51.202526 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:52.901371 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/xan/jbddqvrxeeqrtdwxevvvqdujlsmnx, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c8ad3826-a8f1-4874-8156-409c7be03c51, upload has finished false.
2024.11.05 19:54:52.904122 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/xan/jbddqvrxeeqrtdwxevvvqdujlsmnx, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c8ad3826-a8f1-4874-8156-409c7be03c51, upload has finished false
2024.11.05 19:54:53.058772 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/xan/jbddqvrxeeqrtdwxevvvqdujlsmnx, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c8ad3826-a8f1-4874-8156-409c7be03c51, upload has finished false
2024.11.05 19:54:53.128149 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 20 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:53.139876 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:43: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce60f4
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
23. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
24. /build/src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp:69: DB::MergedColumnOnlyOutputStream::fillChecksums(std::shared_ptr<DB::IMergeTreeDataPart>&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3fe1a
25. /build/src/Storages/MergeTree/MergeTask.cpp:657: DB::MergeTask::VerticalMergeStage::finalizeVerticalMergeForOneColumn() const @ 0x000000001cb6fabd
26. /build/src/Storages/MergeTree/MergeTask.cpp:877: DB::MergeTask::VerticalMergeStage::executeVerticalMergeForAllColumns() const @ 0x000000001cb72f6e
27. /build/src/Storages/MergeTree/MergeTask.cpp:829: DB::MergeTask::VerticalMergeStage::execute() @ 0x000000001cb72cb8
28. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
29. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:54:56.453200 [ 5794 ] {6395dea5-813b-4eef-b0d3-6b7f33b3f21b} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:38402) (comment: 00074_full_join.sql) (in query: SELECT CounterID, hits, visits FROM ( SELECT (CounterID % 100000) AS CounterID, count() AS hits FROM test.hits GROUP BY CounterID ) ANY FULL OUTER JOIN ( SELECT (CounterID % 100000) AS CounterID, sum(Sign) AS visits FROM test.visits GROUP BY CounterID HAVING visits > 0 ) USING CounterID WHERE hits = 0 OR visits = 0 ORDER BY hits + visits * 10 DESC, CounterID ASC LIMIT 20;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:54:57.741652 [ 5794 ] {6395dea5-813b-4eef-b0d3-6b7f33b3f21b} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:54:57.992550 [ 7051 ] {c4ea2bb4-6b3e-4684-b854-30b8c8c9ad54} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=4, jobs=4). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47852) (comment: 02125_lz4_compression_bug_JSONCompactEachRow.sh) (in query: INSERT INTO file SELECT * FROM numbers(2500000)), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:49: DB::threadFunction(DB::CompletedPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d51351d
13. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:90: DB::CompletedPipelineExecutor::execute()::$_0::operator()() const @ 0x000000001d5133aa
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d513355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&>(DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d513339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&>(DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&) @ 0x000000001d51329d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&)::'lambda'()::operator()() @ 0x000000001d5131a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d513115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d5130f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5130d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5130a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:54:58.023156 [ 7051 ] {c4ea2bb4-6b3e-4684-b854-30b8c8c9ad54} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=4, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:49: DB::threadFunction(DB::CompletedPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d51351d
13. /build/src/Processors/Executors/CompletedPipelineExecutor.cpp:90: DB::CompletedPipelineExecutor::execute()::$_0::operator()() const @ 0x000000001d5133aa
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d513355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&>(DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d513339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&>(DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&) @ 0x000000001d51329d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&)::'lambda'()::operator()() @ 0x000000001d5131a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d513115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d5130f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5130d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5130a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:54:58.585979 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {86a06611-889e-455b-b452-b6a6203be276::202411_1_85_17}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:01.635102 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {86a06611-889e-455b-b452-b6a6203be276::202411_1_85_17}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
25. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
26. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
27. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
28. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:02.947582 [ 5215 ] {36524c05-949a-4789-ae60-9200ad38b8ae} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=9, jobs=9). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:54848) (comment: 02020_exponential_smoothing_cross_block.sql) (in query: SELECT countIf(1) FROM (SELECT NULL, exponentialTimeDecayedSum(100000002004087730000.)(value, time) OVER (Rows BETWEEN 255 FOLLOWING AND UNBOUNDED FOLLOWING), number = -2147483649, exponentialTimeDecayedSum(-1)(value, time) OVER (Rows BETWEEN UNBOUNDED PRECEDING AND 65537 PRECEDING), number AS value, number = NULL, exponentialTimeDecayedSum(65537)(value, time) OVER (Rows BETWEEN UNBOUNDED PRECEDING AND 1048577 FOLLOWING), number AS time FROM numbers(65535)) ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:55:03.012745 [ 5215 ] {36524c05-949a-4789-ae60-9200ad38b8ae} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=9, jobs=9). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:55:03.728895 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:04.200447 [ 7051 ] {bd711f42-a9de-4cba-992a-e26b14b27bc6} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:54864) (comment: 02125_lz4_compression_bug_JSONCompactEachRow.sh) (in query: SELECT max(x) FROM file), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:55:04.408334 [ 7051 ] {bd711f42-a9de-4cba-992a-e26b14b27bc6} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:55:04.774218 [ 5215 ] {728d6c58-9585-47de-b5b2-ba0dc6911744} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:54878) (comment: 00143_transform_non_const_default.sql) (in query: SELECT transform(SearchEngineID, [2, 3], ['Яндекс', 'Google'], PageCharset) AS title, count() AS c FROM test.hits WHERE SearchEngineID != 0 GROUP BY title HAVING c > 0 ORDER BY c DESC LIMIT 10;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:55:04.842239 [ 5215 ] {728d6c58-9585-47de-b5b2-ba0dc6911744} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:55:13.951648 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/sik/visxucqlfbhfofituujmhonvxtrak, total size 251658240, count 251658240, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 124c3f7c-1d62-4387-8583-626f78723e3c, upload has finished false.
2024.11.05 19:55:13.953256 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/sik/visxucqlfbhfofituujmhonvxtrak, total size 251658240, count 251658240, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 124c3f7c-1d62-4387-8583-626f78723e3c, upload has finished false
2024.11.05 19:55:14.043030 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/sik/visxucqlfbhfofituujmhonvxtrak, total size 251658240, count 251658240, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 124c3f7c-1d62-4387-8583-626f78723e3c, upload has finished false
2024.11.05 19:55:14.103302 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 54 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:15.525421 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:17.620103 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:18.970818 [ 5795 ] {108745a2-a1f5-4151-a17b-40783bdecb22} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=14, jobs=14). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46814) (comment: 02815_range_dict_no_direct_join.sql) (in query: SELECT id, amount FROM ids INNER JOIN discounts_dict ON id = advertiser_id ORDER BY id, amount SETTINGS join_algorithm = 'direct,hash';), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:55:19.095510 [ 5795 ] {108745a2-a1f5-4151-a17b-40783bdecb22} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=14, jobs=14). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:55:22.183046 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/slh/ozbnncscvubdkzgxjruqvzeckwkrw, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id a904c6e9-8f11-44e3-87fd-094be54431f4, upload has finished false.
2024.11.05 19:55:22.188531 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/slh/ozbnncscvubdkzgxjruqvzeckwkrw, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id a904c6e9-8f11-44e3-87fd-094be54431f4, upload has finished false
2024.11.05 19:55:22.221347 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/slh/ozbnncscvubdkzgxjruqvzeckwkrw, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id a904c6e9-8f11-44e3-87fd-094be54431f4, upload has finished false
2024.11.05 19:55:22.261334 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 15 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:25.989802 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {c04f5a1b-d832-4de4-96b0-dec4e591ea83::202411_1_101_20}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:28.367726 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {c04f5a1b-d832-4de4-96b0-dec4e591ea83::202411_1_101_20}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
24. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
25. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
26. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
27. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:30.246846 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/uxe/sflgaoporpzsysgjxrbvqgfftdojz, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 1e887691-44ba-4b9a-92ad-74f40694790f, upload has finished false.
2024.11.05 19:55:30.249547 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/uxe/sflgaoporpzsysgjxrbvqgfftdojz, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 1e887691-44ba-4b9a-92ad-74f40694790f, upload has finished false
2024.11.05 19:55:30.289763 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/uxe/sflgaoporpzsysgjxrbvqgfftdojz, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 1e887691-44ba-4b9a-92ad-74f40694790f, upload has finished false
2024.11.05 19:55:30.461789 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 19 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:32.923709 [ 5793 ] {b471e23c-08b6-4514-98b9-63f291b69dd8} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:33862) (comment: 00001_count_hits.sql) (in query: SELECT count() FROM test.hits ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:55:33.004857 [ 5793 ] {b471e23c-08b6-4514-98b9-63f291b69dd8} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=6, jobs=6). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:55:33.753514 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:34.566321 [ 5795 ] {394788df-5f88-48e4-b04c-a6a4945677ad} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60772) (comment: 00093_prewhere_array_join.sql) (in query: SELECT arrayJoin([SearchEngineID]) AS search_engine, URL FROM test.hits WHERE SearchEngineID != 0 AND search_engine != 0 FORMAT Null;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:55:35.341992 [ 5795 ] {394788df-5f88-48e4-b04c-a6a4945677ad} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:55:36.046857 [ 3642 ] {} <Error> void DB::SystemLog<DB::MetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::MetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:37.781243 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/mob/bkaomwfuvgowtffdaadnffmkigobw, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id aa6eb525-b686-40d3-8f65-387a568a529a, upload has finished false.
2024.11.05 19:55:37.783080 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/mob/bkaomwfuvgowtffdaadnffmkigobw, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id aa6eb525-b686-40d3-8f65-387a568a529a, upload has finished false
2024.11.05 19:55:37.816438 [ 3573 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/mob/bkaomwfuvgowtffdaadnffmkigobw, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id aa6eb525-b686-40d3-8f65-387a568a529a, upload has finished false
2024.11.05 19:55:37.880462 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 17 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:38.144623 [ 5215 ] {aca8e002-ba28-406d-a0ca-ea8c42d92ece} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=12, jobs=12). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:33832) (comment: 02343_group_by_use_nulls.sql) (in query: SELECT number, number % 2, sum(number) AS val FROM numbers(10) GROUP BY CUBE(number, number % 2) ORDER BY (number, number % 2, val) SETTINGS group_by_use_nulls=1;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:55:38.180065 [ 5215 ] {aca8e002-ba28-406d-a0ca-ea8c42d92ece} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=12, jobs=12). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:55:38.211444 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0aef53a4-984b-414b-bb72-6836bf38bb48::202411_1_114_23}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:38.369616 [ 3613 ] {} <Warning> system.trace_log (4ff3d7e3-1bf7-4d39-8a23-c88787b766d0): Removing temporary directory /var/lib/clickhouse/disks/s3/store/4ff/4ff3d7e3-1bf7-4d39-8a23-c88787b766d0/tmp_merge_202411_1_111_20/
2024.11.05 19:55:38.370241 [ 3613 ] {} <Warning> system.trace_log (4ff3d7e3-1bf7-4d39-8a23-c88787b766d0): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3/store/4ff/4ff3d7e3-1bf7-4d39-8a23-c88787b766d0/tmp_merge_202411_1_111_20/
2024.11.05 19:55:38.520980 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_111_20}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=12). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:43: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce60f4
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
24. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
25. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
26. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
27. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
28. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:39.510306 [ 5215 ] {8bf1fb9b-a19e-484c-81ed-96976cb7112d} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:60780) (comment: 00014_filtering_arrays.sql) (in query: SELECT GeneralInterests FROM test.hits WHERE AdvEngineID != 0 ORDER BY GeneralInterests DESC LIMIT 10 ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:55:39.645679 [ 5215 ] {8bf1fb9b-a19e-484c-81ed-96976cb7112d} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 19:55:39.685782 [ 3643 ] {} <Error> void DB::SystemLog<DB::SessionLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::SessionLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:41.111241 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_111_20}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=1): (while reading column event_date): (while reading from part /var/lib/clickhouse/disks/s3/store/4ff/4ff3d7e3-1bf7-4d39-8a23-c88787b766d0/202411_1_100_19/ in table system.trace_log (4ff3d7e3-1bf7-4d39-8a23-c88787b766d0) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 8192): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:41.985283 [ 5794 ] {22770} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=5): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46822) (comment: 02789_filesystem_cache_alignment.sh) (in query: SELECT * FROM test WHERE NOT ignore() LIMIT 1 FORMAT Null;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:55:42.287087 [ 5794 ] {22770} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=5): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:55:43.745217 [ 7291 ] {} <Error> void DB::AsyncLoader::worker(Pool &): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1011, jobs=900). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:43.773144 [ 4569 ] {} <Error> DDLWorker(test_eg7g8ozy): Error on initialization of test_eg7g8ozy: Code: 695. DB::Exception: Load job 'startup Replicated database test_eg7g8ozy' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1011, jobs=900). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:46.403865 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/sqg/ianlahteddawpfjanpqkriaaiqzwh, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 9d967fcf-371e-49cd-8ab9-70e28af99695, upload has finished false.
2024.11.05 19:55:46.406784 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/sqg/ianlahteddawpfjanpqkriaaiqzwh, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 9d967fcf-371e-49cd-8ab9-70e28af99695, upload has finished false
2024.11.05 19:55:46.459223 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/sqg/ianlahteddawpfjanpqkriaaiqzwh, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 9d967fcf-371e-49cd-8ab9-70e28af99695, upload has finished false
2024.11.05 19:55:46.575035 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=1): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 22 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:48.886711 [ 4569 ] {} <Error> DDLWorker(test_eg7g8ozy): Error on initialization of test_eg7g8ozy: Code: 695. DB::Exception: Load job 'startup Replicated database test_eg7g8ozy' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1011, jobs=900). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:49.005120 [ 3800 ] {7ef46732-8888-42d8-9c47-dbfb42357039} <Error> executeQuery: Code: 107. DB::Exception: File /var/lib/clickhouse/user_files/nonexist.txt doesn't exist. (FILE_DOESNT_EXIST) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:35224) (in query: select _path,_file from file('nonexist.txt', 'CSV', 'val1 char')), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Storages/StorageFile.cpp:1573: DB::StorageFile::read(DB::QueryPlan&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::SelectQueryInfo&, std::shared_ptr<DB::Context const>, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000001c37b16d
7. /build/src/Planner/PlannerJoinTree.cpp:878: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr<DB::IQueryTreeNode>, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::PlannerContext>&, bool, bool) @ 0x000000001b553818
8. /build/src/Planner/PlannerJoinTree.cpp:1667: DB::buildJoinTreeQueryPlan(std::shared_ptr<DB::IQueryTreeNode> const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>> const&, std::shared_ptr<DB::PlannerContext>&) @ 0x000000001b54ef10
9. /build/src/Planner/Planner.cpp:1473: DB::Planner::buildPlanForQueryNode() @ 0x000000001b50fd06
10. /build/src/Planner/Planner.cpp:0: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000001b50be55
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:234: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000001b50905d
12. /build/src/Interpreters/executeQuery.cpp:1146: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2812
13. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
14. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
15. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:55:49.014688 [ 3800 ] {7ef46732-8888-42d8-9c47-dbfb42357039} <Error> TCPHandler: Code: 107. DB::Exception: File /var/lib/clickhouse/user_files/nonexist.txt doesn't exist. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Storages/StorageFile.cpp:1573: DB::StorageFile::read(DB::QueryPlan&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::SelectQueryInfo&, std::shared_ptr<DB::Context const>, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000001c37b16d
7. /build/src/Planner/PlannerJoinTree.cpp:878: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr<DB::IQueryTreeNode>, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::PlannerContext>&, bool, bool) @ 0x000000001b553818
8. /build/src/Planner/PlannerJoinTree.cpp:1667: DB::buildJoinTreeQueryPlan(std::shared_ptr<DB::IQueryTreeNode> const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>> const&, std::shared_ptr<DB::PlannerContext>&) @ 0x000000001b54ef10
9. /build/src/Planner/Planner.cpp:1473: DB::Planner::buildPlanForQueryNode() @ 0x000000001b50fd06
10. /build/src/Planner/Planner.cpp:0: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000001b50be55
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:234: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000001b50905d
12. /build/src/Interpreters/executeQuery.cpp:1146: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2812
13. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
14. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
15. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:55:49.115540 [ 5215 ] {0d56b29b-2df1-4c11-8559-3a0e7bc7bed8} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=3): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:35198) (comment: 00165_jit_aggregate_functions.sql) (in query: SELECT CounterID, min(WatchID), max(WatchID), sum(WatchID), avg(WatchID), avgWeighted(WatchID, CounterID), count(WatchID), groupBitOr(WatchID), groupBitAnd(WatchID), groupBitXor(WatchID) FROM test.hits GROUP BY CounterID ORDER BY count() DESC LIMIT 20;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:55:49.116038 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 2 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:49.932818 [ 5215 ] {0d56b29b-2df1-4c11-8559-3a0e7bc7bed8} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=3): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:55:51.752599 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 3 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:53.923437 [ 4569 ] {} <Error> DDLWorker(test_eg7g8ozy): Error on initialization of test_eg7g8ozy: Code: 695. DB::Exception: Load job 'startup Replicated database test_eg7g8ozy' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1011, jobs=900). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:54.799907 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 3 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:55:57.211205 [ 5793 ] {a431d02b-09d8-4730-a475-69dd7d8d8ac3} <Error> executeQuery: Code: 107. DB::Exception: File /var/lib/clickhouse/user_files/nonexist.txt doesn't exist. (FILE_DOESNT_EXIST) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:54190) (in query: select _path,_file from file('nonexist.txt', 'CSV', 'val1 char')), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Storages/StorageFile.cpp:1573: DB::StorageFile::read(DB::QueryPlan&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::SelectQueryInfo&, std::shared_ptr<DB::Context const>, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000001c37b16d
7. /build/src/Planner/PlannerJoinTree.cpp:878: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr<DB::IQueryTreeNode>, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::PlannerContext>&, bool, bool) @ 0x000000001b553818
8. /build/src/Planner/PlannerJoinTree.cpp:1667: DB::buildJoinTreeQueryPlan(std::shared_ptr<DB::IQueryTreeNode> const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>> const&, std::shared_ptr<DB::PlannerContext>&) @ 0x000000001b54ef10
9. /build/src/Planner/Planner.cpp:1473: DB::Planner::buildPlanForQueryNode() @ 0x000000001b50fd06
10. /build/src/Planner/Planner.cpp:0: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000001b50be55
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:234: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000001b50905d
12. /build/src/Interpreters/executeQuery.cpp:1146: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2812
13. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
14. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
15. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:55:57.215881 [ 5793 ] {a431d02b-09d8-4730-a475-69dd7d8d8ac3} <Error> TCPHandler: Code: 107. DB::Exception: File /var/lib/clickhouse/user_files/nonexist.txt doesn't exist. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000a943833
6. /build/src/Storages/StorageFile.cpp:1573: DB::StorageFile::read(DB::QueryPlan&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::SelectQueryInfo&, std::shared_ptr<DB::Context const>, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000001c37b16d
7. /build/src/Planner/PlannerJoinTree.cpp:878: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr<DB::IQueryTreeNode>, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::PlannerContext>&, bool, bool) @ 0x000000001b553818
8. /build/src/Planner/PlannerJoinTree.cpp:1667: DB::buildJoinTreeQueryPlan(std::shared_ptr<DB::IQueryTreeNode> const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>> const&, std::shared_ptr<DB::PlannerContext>&) @ 0x000000001b54ef10
9. /build/src/Planner/Planner.cpp:1473: DB::Planner::buildPlanForQueryNode() @ 0x000000001b50fd06
10. /build/src/Planner/Planner.cpp:0: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000001b50be55
11. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:234: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000001b50905d
12. /build/src/Interpreters/executeQuery.cpp:1146: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2812
13. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
14. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
15. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:55:58.945987 [ 4569 ] {} <Error> DDLWorker(test_eg7g8ozy): Error on initialization of test_eg7g8ozy: Code: 695. DB::Exception: Load job 'startup Replicated database test_eg7g8ozy' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1011, jobs=900). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:01.394316 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/qxg/xwgsmlsvamkljpmijdraugrrmxetq, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id fc17d272-feb8-406d-abb4-3cc0c6981ef8, upload has finished false.
2024.11.05 19:56:01.397496 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/qxg/xwgsmlsvamkljpmijdraugrrmxetq, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id fc17d272-feb8-406d-abb4-3cc0c6981ef8, upload has finished false
2024.11.05 19:56:01.424483 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/qxg/xwgsmlsvamkljpmijdraugrrmxetq, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id fc17d272-feb8-406d-abb4-3cc0c6981ef8, upload has finished false
2024.11.05 19:56:01.556764 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 14 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:01.824117 [ 3800 ] {1dd4ba38-84e9-42f0-bd3a-bf5a4c039194} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:54220) (comment: 02026_storage_filelog_largefile.sh) (in query: insert into function file('/var/lib/clickhouse/user_files/02026_storage_filelog_largefile_test_3/test1.csv', 'CSV', 'k UInt32, v UInt32') select number, number from numbers(10000);), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.h:298: DB::ParallelFormattingOutputFormat::scheduleFormatterThreadForUnitWithNumber(unsigned long, unsigned long) @ 0x000000001d7bfaf0
10. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.cpp:94: DB::ParallelFormattingOutputFormat::addChunk(DB::Chunk, DB::ParallelFormattingOutputFormat::ProcessingUnitType, bool) @ 0x000000001d7be7e8
11. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.h:121: DB::ParallelFormattingOutputFormat::writePrefix() @ 0x000000001d7c03c4
12. /build/src/Processors/Formats/IOutputFormat.h:103: DB::IOutputFormat::writePrefixIfNeeded() @ 0x000000000d8378d3
13. /build/src/Processors/Formats/IOutputFormat.cpp:116: DB::IOutputFormat::write(DB::Block const&) @ 0x000000001d5483e0
14. /build/src/Storages/StorageFile.cpp:1778: DB::StorageFileSink::consume(DB::Chunk) @ 0x000000001c3916ab
15. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
16. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:114: DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x000000001da41c5a
24. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work() @ 0x000000001da4169e
25. /build/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001d5343e3
26. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001d534120
27. /build/src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d5166a1
28. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
29. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915

2024.11.05 19:56:01.883284 [ 3800 ] {1dd4ba38-84e9-42f0-bd3a-bf5a4c039194} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=0, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.h:298: DB::ParallelFormattingOutputFormat::scheduleFormatterThreadForUnitWithNumber(unsigned long, unsigned long) @ 0x000000001d7bfaf0
10. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.cpp:94: DB::ParallelFormattingOutputFormat::addChunk(DB::Chunk, DB::ParallelFormattingOutputFormat::ProcessingUnitType, bool) @ 0x000000001d7be7e8
11. /build/src/Processors/Formats/Impl/ParallelFormattingOutputFormat.h:121: DB::ParallelFormattingOutputFormat::writePrefix() @ 0x000000001d7c03c4
12. /build/src/Processors/Formats/IOutputFormat.h:103: DB::IOutputFormat::writePrefixIfNeeded() @ 0x000000000d8378d3
13. /build/src/Processors/Formats/IOutputFormat.cpp:116: DB::IOutputFormat::write(DB::Block const&) @ 0x000000001d5483e0
14. /build/src/Storages/StorageFile.cpp:1778: DB::StorageFileSink::consume(DB::Chunk) @ 0x000000001c3916ab
15. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
16. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
23. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:114: DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x000000001da41c5a
24. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work() @ 0x000000001da4169e
25. /build/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001d5343e3
26. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001d534120
27. /build/src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d5166a1
28. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
29. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915

2024.11.05 19:56:03.972775 [ 4569 ] {} <Error> DDLWorker(test_eg7g8ozy): Error on initialization of test_eg7g8ozy: Code: 695. DB::Exception: Load job 'startup Replicated database test_eg7g8ozy' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1011, jobs=900). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:06.583468 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:08.105473 [ 3580 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/cgu/ojwodgotiaymduoifxtdheinutfvu, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 83437420-4b4a-4042-86e8-f93f22709bcf, upload has finished false.
2024.11.05 19:56:08.107868 [ 3580 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/cgu/ojwodgotiaymduoifxtdheinutfvu, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 83437420-4b4a-4042-86e8-f93f22709bcf, upload has finished false
2024.11.05 19:56:08.161166 [ 3580 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/cgu/ojwodgotiaymduoifxtdheinutfvu, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 83437420-4b4a-4042-86e8-f93f22709bcf, upload has finished false
2024.11.05 19:56:08.275784 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 15 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:09.011313 [ 4569 ] {} <Error> DDLWorker(test_eg7g8ozy): Error on initialization of test_eg7g8ozy: Code: 695. DB::Exception: Load job 'startup Replicated database test_eg7g8ozy' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1011, jobs=900). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:10.896634 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 4 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:11.049581 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:11.080543 [ 3641 ] {} <Error> void DB::SystemLog<DB::PartLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::PartLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:12.108347 [ 3580 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key s3/qvj/ciyhpfdyrelnseseidqpeqjjjjuak, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c2b3e600-bc29-42df-b820-6450b79c58e7, upload has finished false.
2024.11.05 19:56:12.111911 [ 3580 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key s3/qvj/ciyhpfdyrelnseseidqpeqjjjjuak, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c2b3e600-bc29-42df-b820-6450b79c58e7, upload has finished false
2024.11.05 19:56:12.169051 [ 3580 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key s3/qvj/ciyhpfdyrelnseseidqpeqjjjjuak, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c2b3e600-bc29-42df-b820-6450b79c58e7, upload has finished false
2024.11.05 19:56:12.509185 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column Referer): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_4_4_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 25 with max_rows_to_read = 6741): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:12.897676 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 3 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:14.097108 [ 4569 ] {} <Error> DDLWorker(test_eg7g8ozy): Error on initialization of test_eg7g8ozy: Code: 695. DB::Exception: Load job 'startup Replicated database test_eg7g8ozy' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1011, jobs=900). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:15.841142 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:17.845013 [ 5793 ] {5cb9232a-1cd4-483b-a1a2-4c6918736ce5} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=24, jobs=24). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52018) (comment: 02163_shard_num.sql) (in query: SELECT _shard_num AS shard_num, sum(1) as rows FROM remote('127.{1,2}', system, one) GROUP BY shard_num ORDER BY shard_num;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:56:17.916624 [ 5793 ] {5cb9232a-1cd4-483b-a1a2-4c6918736ce5} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=24, jobs=24). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:56:18.671308 [ 5794 ] {b0404c12-4e24-4fd2-b290-612950c36a2f} <Error> executeQuery: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:59924). (NETWORK_ERROR) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59924) (comment: 02163_shard_num.sql) (in query: SELECT shardNum() AS `shard_num`, count() AS `rows` FROM `system`.`one` AS `__table1` GROUP BY shardNum() ORDER BY shardNum() ASC), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:169: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e9244
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Server/TCPHandler.cpp:2217: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001d47db8f
9. /build/src/Server/TCPHandler.cpp:1084: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b9aa
10. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:56:18.675885 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_117_21}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=11). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/SerializationSparse.cpp:165: DB::SerializationSparse::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x000000001a052e89
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:18.758713 [ 5794 ] {b0404c12-4e24-4fd2-b290-612950c36a2f} <Error> TCPHandler: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:59924). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:169: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e9244
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Server/TCPHandler.cpp:2217: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001d47db8f
9. /build/src/Server/TCPHandler.cpp:1084: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b9aa
10. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:56:18.768388 [ 5794 ] {b0404c12-4e24-4fd2-b290-612950c36a2f} <Warning> TCPHandler: Client has gone away.
2024.11.05 19:56:19.133398 [ 4569 ] {} <Error> DDLWorker(test_eg7g8ozy): Error on initialization of test_eg7g8ozy: Code: 695. DB::Exception: Load job 'startup Replicated database test_eg7g8ozy' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1011, jobs=900). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:19.370460 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=1): (while reading column UserID): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_4_4_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 0 with max_rows_to_read = 6741): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:22.840142 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:22.972071 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/sub/pjecnqmgeutysxactjawgdizhppej, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c46eb989-8a1c-40ce-894f-9ce50bbf09b7, upload has finished false.
2024.11.05 19:56:22.978504 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/sub/pjecnqmgeutysxactjawgdizhppej, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c46eb989-8a1c-40ce-894f-9ce50bbf09b7, upload has finished false
2024.11.05 19:56:23.037242 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/sub/pjecnqmgeutysxactjawgdizhppej, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c46eb989-8a1c-40ce-894f-9ce50bbf09b7, upload has finished false
2024.11.05 19:56:23.140840 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 25 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:24.152546 [ 4569 ] {} <Error> DDLWorker(test_eg7g8ozy): Error on initialization of test_eg7g8ozy: Code: 695. DB::Exception: Load job 'startup Replicated database test_eg7g8ozy' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1011, jobs=900). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:24.167950 [ 5215 ] {db08e923-3b76-42d8-9cb0-7e4c4f65ae2a} <Error> executeQuery: Code: 306. DB::Exception: Maximum parse depth (1000) exceeded. Consider rising max_parser_depth parameter. (TOO_DEEP_RECURSION) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35300) (comment: 00984_parser_stack_overflow.sh) (in query: ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<unsigned int&>(int, FormatStringHelperImpl<std::type_identity<unsigned int&>::type>, unsigned int&) @ 0x00000000191464d3
6. /build/src/Parsers/IParser.h:82: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384f46
7. /build/src/Parsers/ExpressionListParsers.cpp:82: DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e35ebb9
8. /build/src/Parsers/ExpressionListParsers.h:89: bool DB::ParserUnionList::parseUtil<DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0, DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_1>(DB::IParser::Pos&, DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0 const&, DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_1 const&) @ 0x000000001e356b40
9. /build/src/Parsers/ExpressionListParsers.cpp:126: DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e3569cb
10. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
11. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
12. /build/src/Parsers/ParserSelectWithUnionQuery.cpp:15: DB::ParserSelectWithUnionQuery::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e3cbf56
13. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
14. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
15. /build/src/Parsers/ExpressionElementParsers.cpp:115: DB::ParserSubquery::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e32e897
16. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
17. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
18. /build/src/Parsers/ParserUnionQueryElement.cpp:13: DB::ParserUnionQueryElement::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e3ecaa9
19. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
20. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
21. /build/src/Parsers/ExpressionListParsers.cpp:82: DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e35ebb9
22. /build/src/Parsers/ExpressionListParsers.h:89: bool DB::ParserUnionList::parseUtil<DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0, DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_1>(DB::IParser::Pos&, DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0 const&, DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_1 const&) @ 0x000000001e356b40
23. /build/src/Parsers/ExpressionListParsers.cpp:126: DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e3569cb
24. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
25. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
26. /build/src/Parsers/ParserSelectWithUnionQuery.cpp:15: DB::ParserSelectWithUnionQuery::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e3cbf56
27. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
28. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
29. /build/src/Parsers/ExpressionElementParsers.cpp:115: DB::ParserSubquery::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e32e897
30. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
31. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba

2024.11.05 19:56:24.278151 [ 5215 ] {db08e923-3b76-42d8-9cb0-7e4c4f65ae2a} <Error> DynamicQueryHandler: Code: 306. DB::Exception: Maximum parse depth (1000) exceeded. Consider rising max_parser_depth parameter. (TOO_DEEP_RECURSION), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<unsigned int&>(int, FormatStringHelperImpl<std::type_identity<unsigned int&>::type>, unsigned int&) @ 0x00000000191464d3
6. /build/src/Parsers/IParser.h:82: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384f46
7. /build/src/Parsers/ExpressionListParsers.cpp:82: DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e35ebb9
8. /build/src/Parsers/ExpressionListParsers.h:89: bool DB::ParserUnionList::parseUtil<DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0, DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_1>(DB::IParser::Pos&, DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0 const&, DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_1 const&) @ 0x000000001e356b40
9. /build/src/Parsers/ExpressionListParsers.cpp:126: DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e3569cb
10. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
11. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
12. /build/src/Parsers/ParserSelectWithUnionQuery.cpp:15: DB::ParserSelectWithUnionQuery::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e3cbf56
13. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
14. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
15. /build/src/Parsers/ExpressionElementParsers.cpp:115: DB::ParserSubquery::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e32e897
16. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
17. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
18. /build/src/Parsers/ParserUnionQueryElement.cpp:13: DB::ParserUnionQueryElement::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e3ecaa9
19. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
20. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
21. /build/src/Parsers/ExpressionListParsers.cpp:82: DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e35ebb9
22. /build/src/Parsers/ExpressionListParsers.h:89: bool DB::ParserUnionList::parseUtil<DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0, DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_1>(DB::IParser::Pos&, DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0 const&, DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_1 const&) @ 0x000000001e356b40
23. /build/src/Parsers/ExpressionListParsers.cpp:126: DB::ParserUnionList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e3569cb
24. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
25. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
26. /build/src/Parsers/ParserSelectWithUnionQuery.cpp:15: DB::ParserSelectWithUnionQuery::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e3cbf56
27. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
28. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
29. /build/src/Parsers/ExpressionElementParsers.cpp:115: DB::ParserSubquery::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e32e897
30. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
31. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:24.428135 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:24.990199 [ 5793 ] {b2adf18c-eff9-43b7-9346-a9326cf33854} <Error> executeQuery: Code: 306. DB::Exception: Maximum parse depth (1000) exceeded. Consider rising max_parser_depth parameter. (TOO_DEEP_RECURSION) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35306) (comment: 00984_parser_stack_overflow.sh) (in query: SELECT [[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<unsigned int&>(int, FormatStringHelperImpl<std::type_identity<unsigned int&>::type>, unsigned int&) @ 0x00000000191464d3
6. /build/src/Parsers/IParser.h:82: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384f46
7. /build/src/Parsers/ExpressionElementParsers.cpp:859: DB::ParserNull::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e33659b
8. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
9. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
10. /build/src/Parsers/ExpressionElementParsers.cpp:1429: DB::ParserLiteral::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e33b171
11. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
12. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
13. /build/src/Parsers/ExpressionElementParsers.cpp:1241: DB::ParserCollectionOfLiterals<DB::Array>::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e34894d
14. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
15. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
16. /build/src/Parsers/ExpressionElementParsers.h:321: DB::ParserArrayOfLiterals::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001da04ff1
17. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
18. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
19. /build/src/Parsers/ExpressionListParsers.cpp:2622: DB::ParserExpressionImpl::tryParseOperand(std::vector<std::unique_ptr<DB::Layer, std::default_delete<DB::Layer>>, std::allocator<std::unique_ptr<DB::Layer, std::default_delete<DB::Layer>>>>&, DB::IParser::Pos&, DB::Expected&) @ 0x000000001e35bf4a
20. /build/src/Parsers/ExpressionListParsers.cpp:2460: DB::ParserExpressionImpl::parse(std::unique_ptr<DB::Layer, std::default_delete<DB::Layer>>, DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e35a307
21. /build/src/Parsers/ExpressionListParsers.cpp:2342: DB::ParserExpression::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e359f2a
22. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
23. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
24. /build/src/Parsers/ExpressionElementParsers.cpp:2005: DB::ParserWithOptionalAlias::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e341064
25. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
26. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
27. /build/src/Parsers/ExpressionListParsers.h:206: DB::ParserExpressionWithOptionalAlias::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e35301a
28. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
29. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
30. /build/src/Parsers/ExpressionListParsers.cpp:53: DB::ParserList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e35eaf1
31. /build/src/Parsers/ExpressionListParsers.h:40: bool DB::ParserList::parseUtil<DB::ParserList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0>(DB::IParser::Pos&, DB::Expected&, DB::ParserList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0 const&, DB::IParser&, bool) @ 0x000000001e3567cc

2024.11.05 19:56:25.019462 [ 5793 ] {b2adf18c-eff9-43b7-9346-a9326cf33854} <Error> DynamicQueryHandler: Code: 306. DB::Exception: Maximum parse depth (1000) exceeded. Consider rising max_parser_depth parameter. (TOO_DEEP_RECURSION), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<unsigned int&>(int, FormatStringHelperImpl<std::type_identity<unsigned int&>::type>, unsigned int&) @ 0x00000000191464d3
6. /build/src/Parsers/IParser.h:82: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384f46
7. /build/src/Parsers/ExpressionElementParsers.cpp:859: DB::ParserNull::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e33659b
8. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
9. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
10. /build/src/Parsers/ExpressionElementParsers.cpp:1429: DB::ParserLiteral::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e33b171
11. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
12. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
13. /build/src/Parsers/ExpressionElementParsers.cpp:1241: DB::ParserCollectionOfLiterals<DB::Array>::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e34894d
14. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
15. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
16. /build/src/Parsers/ExpressionElementParsers.h:321: DB::ParserArrayOfLiterals::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001da04ff1
17. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
18. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
19. /build/src/Parsers/ExpressionListParsers.cpp:2622: DB::ParserExpressionImpl::tryParseOperand(std::vector<std::unique_ptr<DB::Layer, std::default_delete<DB::Layer>>, std::allocator<std::unique_ptr<DB::Layer, std::default_delete<DB::Layer>>>>&, DB::IParser::Pos&, DB::Expected&) @ 0x000000001e35bf4a
20. /build/src/Parsers/ExpressionListParsers.cpp:2460: DB::ParserExpressionImpl::parse(std::unique_ptr<DB::Layer, std::default_delete<DB::Layer>>, DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e35a307
21. /build/src/Parsers/ExpressionListParsers.cpp:2342: DB::ParserExpression::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e359f2a
22. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
23. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
24. /build/src/Parsers/ExpressionElementParsers.cpp:2005: DB::ParserWithOptionalAlias::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e341064
25. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
26. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
27. /build/src/Parsers/ExpressionListParsers.h:206: DB::ParserExpressionWithOptionalAlias::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e35301a
28. /build/src/Parsers/IParserBase.cpp:13: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e385109
29. /build/src/Parsers/IParserBase.h:31: DB::IParserBase::parse(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&) @ 0x000000001e384fba
30. /build/src/Parsers/ExpressionListParsers.cpp:53: DB::ParserList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0::operator()() const @ 0x000000001e35eaf1
31. /build/src/Parsers/ExpressionListParsers.h:40: bool DB::ParserList::parseUtil<DB::ParserList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0>(DB::IParser::Pos&, DB::Expected&, DB::ParserList::parseImpl(DB::IParser::Pos&, std::shared_ptr<DB::IAST>&, DB::Expected&)::$_0 const&, DB::IParser&, bool) @ 0x000000001e3567cc
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:26.079785 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_117_21}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:43: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce60f4
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
24. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
25. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
26. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
27. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
28. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:27.085258 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 5 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:29.217886 [ 4569 ] {} <Error> DDLWorker(test_eg7g8ozy): Error on initialization of test_eg7g8ozy: Code: 695. DB::Exception: Load job 'startup Replicated database test_eg7g8ozy' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1011, jobs=900). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:29.887153 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_117_21}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column ptr): (while reading from part /var/lib/clickhouse/disks/s3/store/4ff/4ff3d7e3-1bf7-4d39-8a23-c88787b766d0/202411_1_111_20/ in table system.trace_log (4ff3d7e3-1bf7-4d39-8a23-c88787b766d0) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 8192): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/SerializationSparse.cpp:165: DB::SerializationSparse::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x000000001a052e89
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:31.784899 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0aef53a4-984b-414b-bb72-6836bf38bb48::202411_1_120_24}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:34.233282 [ 4569 ] {} <Error> DDLWorker(test_eg7g8ozy): Error on initialization of test_eg7g8ozy: Code: 695. DB::Exception: Load job 'startup Replicated database test_eg7g8ozy' failed: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1011, jobs=900). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a665eb3
10. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<ThreadFromGlobalPoolImpl<true>>::__unique_single std::make_unique[abi:v15000]<ThreadFromGlobalPoolImpl<true>, void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&) @ 0x000000001a6638d1
11. /build/src/Interpreters/DDLWorker.cpp:131: DB::DDLWorker::startup() @ 0x000000001a6449f7
12. /build/src/Databases/DatabaseReplicated.cpp:578: DB::DatabaseReplicated::startupDatabaseAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a11015a
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a10f6a5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a10f665
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a10f625
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a10f5ed
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
19. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
20. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
21. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
28. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
29. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
31. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:942: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f495d
7. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
9. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
14. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
15. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
17. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
18. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
19. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
20. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
26. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
27. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
29. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
30. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:35.754173 [ 5795 ] {3254201e-24d0-4b88-b3d6-478cd4c076bf} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:35214) (comment: 00041_aggregating_materialized_view.sql) (in query: SELECT StartDate, sum(Sign) AS Visits, uniq(UserID) AS Users FROM test.visits WHERE CounterID = 942285 GROUP BY StartDate ORDER BY StartDate;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:56:35.772330 [ 5795 ] {3254201e-24d0-4b88-b3d6-478cd4c076bf} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:748: DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>&&, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>&&, std::shared_ptr<DB::StorageInMemoryMetadata const>, std::shared_ptr<DB::Context const> const&, DB::KeyCondition const&, std::optional<DB::KeyCondition> const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, std::shared_ptr<Poco::Logger>, unsigned long, std::vector<DB::ReadFromMergeTree::IndexStat, std::allocator<DB::ReadFromMergeTree::IndexStat>>&, bool) @ 0x000000001cd10935
10. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1671: DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5f3c4
11. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1570: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::shared_ptr<DB::Context const>, unsigned long, std::shared_ptr<std::unordered_map<String, long, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, long>>>>, DB::MergeTreeData const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<Poco::Logger>, std::optional<DB::ReadFromMergeTree::Indexes>&) @ 0x000000001dd5c06f
12. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1371: DB::ReadFromMergeTree::selectRangesToRead(std::vector<std::shared_ptr<DB::IMergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::IMergeTreeDataPart const>>>, std::vector<std::shared_ptr<DB::AlterConversions const>, std::allocator<std::shared_ptr<DB::AlterConversions const>>>) const @ 0x000000001dd5be9e
13. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1854: DB::ReadFromMergeTree::getAnalysisResult() const @ 0x000000001dd61c8c
14. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1969: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd62cfb
15. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd09490
16. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001dd33f1f
17. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001b508f41
18. /build/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001b508c9b
19. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
20. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
21. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
22. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
23. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
24. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
25. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
26. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
27. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
28. ? @ 0x00007f3c91188ac3
29. ? @ 0x00007f3c9121a850

2024.11.05 19:56:45.604464 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/tuq/hirvddhavslnwgylyywvunowulnco, total size 268435456, count 268435456, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 9a17f8bf-b8a5-44fc-9d73-9ececafcb2c8, upload has finished false.
2024.11.05 19:56:45.605908 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/tuq/hirvddhavslnwgylyywvunowulnco, total size 268435456, count 268435456, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 9a17f8bf-b8a5-44fc-9d73-9ececafcb2c8, upload has finished false
2024.11.05 19:56:45.691531 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/tuq/hirvddhavslnwgylyywvunowulnco, total size 268435456, count 268435456, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 9a17f8bf-b8a5-44fc-9d73-9ececafcb2c8, upload has finished false
2024.11.05 19:56:45.702504 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:567: DB::WriteBufferFromS3::writePart(DB::WriteBufferFromS3::PartData&&) @ 0x0000000019914d6e
18. /build/src/IO/WriteBufferFromS3.cpp:370: DB::WriteBufferFromS3::writeMultipartUpload() @ 0x0000000019908eb6
19. /build/src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::nextImpl() @ 0x000000001990857e
20. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
21. /build/src/IO/WriteBufferFromFileDecorator.cpp:71: DB::WriteBufferFromFileDecorator::nextImpl() @ 0x000000001a3593ad
22. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
23. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
24. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
25. /build/src/IO/WriteBuffer.h:76: DB::WriteBuffer::nextIfAtEnd() @ 0x000000000a931978
26. /build/src/IO/WriteBuffer.h:91: DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000a9317e0
27. /build/src/Compression/CompressedWriteBuffer.cpp:58: DB::CompressedWriteBuffer::nextImpl() @ 0x0000000019ddd7ed
28. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
29. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
30. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
31. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:350: DB::MergeTreeDataPartWriterWide::getCurrentMarksForColumn(DB::NameAndTypePair const&, std::set<String, std::less<String>, std::allocator<String>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ccf5839
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:46.005404 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:47.674104 [ 3574 ] {} <Error> 46fb1e3e-aec4-44c2-b6db-96b57d25277c::all_0_0_1 (MergeFromLogEntryTask): virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
24. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
25. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
26. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
27. /build/src/Storages/MergeTree/MergeFromLogEntryTask.h:36: DB::MergeFromLogEntryTask::executeInnerTask() @ 0x000000001cb5d384
28. /build/src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:208: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000001cfea799
29. /build/src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:47: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000001cfe8980
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:47.678570 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {46fb1e3e-aec4-44c2-b6db-96b57d25277c::all_0_0_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
24. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
25. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
26. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
27. /build/src/Storages/MergeTree/MergeFromLogEntryTask.h:36: DB::MergeFromLogEntryTask::executeInnerTask() @ 0x000000001cb5d384
28. /build/src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:208: DB::ReplicatedMergeMutateTaskBase::executeImpl() @ 0x000000001cfea799
29. /build/src/Storages/MergeTree/ReplicatedMergeMutateTaskBase.cpp:47: DB::ReplicatedMergeMutateTaskBase::executeStep() @ 0x000000001cfe8980
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:48.004541 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:48.569008 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 6 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:51.132455 [ 7051 ] {1ec4edbf-fdc9-45f6-96b9-de080c184023} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48376) (comment: 00161_parallel_parsing_with_names.sh) (in query: SELECT URLRegions as d, toTimeZone(ClientEventTime, 'Asia/Dubai') as a, MobilePhoneModel as b, ParamPrice as e, ClientIP6 as c FROM test.hits LIMIT 50000 Format CSVWithNames), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/SerializationArray.cpp:248: DB::SerializationArray::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fe0c37

2024.11.05 19:56:51.166699 [ 7051 ] {1ec4edbf-fdc9-45f6-96b9-de080c184023} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/SerializationArray.cpp:248: DB::SerializationArray::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fe0c37

2024.11.05 19:56:51.868399 [ 3800 ] {c5fa2a13-e34a-4cdf-b6a7-f1dcf4cb2006} <Error> executeQuery: Code: 117. DB::Exception: Incorrect data: is_deleted = 2 (must be 1 or 0). (INCORRECT_DATA) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52026) (comment: 02490_replacing_merge_tree_is_deleted_column.sql) (in query: -- is_deleted == 0/1
 INSERT INTO test (*) VALUES ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Processors/Merges/Algorithms/ReplacingSortedAlgorithm.cpp:141: DB::ReplacingSortedAlgorithm::merge() @ 0x000000001dc617e4
7. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:383: DB::MergeTreeDataWriter::mergeBlock(DB::Block&&, DB::SortDescription, std::vector<String, std::allocator<String>> const&, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul>*&, DB::MergeTreeData::MergingParams const&) @ 0x000000001cf83479
8. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:505: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf84dc4
9. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
10. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
11. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
12. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:114: DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x000000001da41c5a
20. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work() @ 0x000000001da4169e
21. /build/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001d5343e3
22. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001d534120
23. /build/src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d5166a1
24. /build/src/Processors/Executors/PipelineExecutor.cpp:148: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000001d51619e
25. /build/src/Processors/Executors/PushingPipelineExecutor.cpp:123: DB::PushingPipelineExecutor::finish() @ 0x000000001d54235e
26. /build/src/Server/TCPHandler.cpp:920: auto DB::TCPHandler::processInsertQuery()::$_0::operator()<DB::PushingPipelineExecutor>(DB::PushingPipelineExecutor&, DB::Block) const @ 0x000000001d47e31e
27. /build/src/Server/TCPHandler.cpp:987: DB::TCPHandler::processInsertQuery() @ 0x000000001d47b22a
28. /build/src/Server/TCPHandler.cpp:534: DB::TCPHandler::runImpl() @ 0x000000001d470731
29. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
30. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
31. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68

2024.11.05 19:56:51.873779 [ 3800 ] {c5fa2a13-e34a-4cdf-b6a7-f1dcf4cb2006} <Error> TCPHandler: Code: 117. DB::Exception: Incorrect data: is_deleted = 2 (must be 1 or 0). (INCORRECT_DATA), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x000000000a8f05f3
6. /build/src/Processors/Merges/Algorithms/ReplacingSortedAlgorithm.cpp:141: DB::ReplacingSortedAlgorithm::merge() @ 0x000000001dc617e4
7. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:383: DB::MergeTreeDataWriter::mergeBlock(DB::Block&&, DB::SortDescription, std::vector<String, std::allocator<String>> const&, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul>*&, DB::MergeTreeData::MergingParams const&) @ 0x000000001cf83479
8. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:505: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf84dc4
9. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
10. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
11. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
12. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:114: DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x000000001da41c5a
20. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work() @ 0x000000001da4169e
21. /build/src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001d5343e3
22. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001d534120
23. /build/src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d5166a1
24. /build/src/Processors/Executors/PipelineExecutor.cpp:148: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000001d51619e
25. /build/src/Processors/Executors/PushingPipelineExecutor.cpp:123: DB::PushingPipelineExecutor::finish() @ 0x000000001d54235e
26. /build/src/Server/TCPHandler.cpp:920: auto DB::TCPHandler::processInsertQuery()::$_0::operator()<DB::PushingPipelineExecutor>(DB::PushingPipelineExecutor&, DB::Block) const @ 0x000000001d47e31e
27. /build/src/Server/TCPHandler.cpp:987: DB::TCPHandler::processInsertQuery() @ 0x000000001d47b22a
28. /build/src/Server/TCPHandler.cpp:534: DB::TCPHandler::runImpl() @ 0x000000001d470731
29. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
30. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
31. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68

2024.11.05 19:56:51.877303 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:52.876679 [ 3800 ] {d6c2e04e-96ee-4077-bb07-49d3a2f76d5f} <Error> executeQuery: Code: 169. DB::Exception: is_deleted column (is_deleted) for storage ReplacingMergeTree must have type UInt8. Provided column of type String. (BAD_TYPE_OF_FIELD) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52026) (comment: 02490_replacing_merge_tree_is_deleted_column.sql) (in query: CREATE TABLE test (`uid` String, `version` UInt32, `is_deleted` String) ENGINE = ReplacingMergeTree(version, is_deleted) ORDER BY uid SETTINGS index_granularity = 16139, min_bytes_for_wide_part = 1073741824, ratio_of_defaults_for_sparse_serialization = 1., replace_long_file_name_to_hash = false, max_file_name_length = 128, min_bytes_for_full_part_storage = 0, compact_parts_max_bytes_to_buffer = 371228609, compact_parts_max_granules_to_buffer = 256, compact_parts_merge_max_bytes_to_prefetch_part = 30980270, merge_max_block_size = 11426, old_parts_lifetime = 392., prefer_fetch_merged_part_size_threshold = 1, vertical_merge_algorithm_min_rows_to_activate = 1000000, vertical_merge_algorithm_min_columns_to_activate = 100, min_merge_bytes_to_use_direct_io = 144654631, index_granularity_bytes = 19865899, concurrent_part_removal_threshold = 91, allow_vertical_merges_from_compact_to_wide_parts = true, cache_populated_by_fetch = false, marks_compress_block_size = 71741, primary_key_compress_block_size = 54237), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String const&, String&&) @ 0x000000001a16d768
6. /build/src/Storages/MergeTree/MergeTreeData.cpp:968: DB::MergeTreeData::MergingParams::check(DB::StorageInMemoryMetadata const&) const::$_2::operator()(bool, String const&) const @ 0x000000001cbb8606
7. /build/src/Storages/MergeTree/MergeTreeData.cpp:1017: DB::MergeTreeData::MergingParams::check(DB::StorageInMemoryMetadata const&) const @ 0x000000001cbb1436
8. /build/src/Storages/MergeTree/MergeTreeData.cpp:401: DB::MergeTreeData::MergeTreeData(DB::StorageID const&, DB::StorageInMemoryMetadata const&, std::shared_ptr<DB::Context>, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, bool, DB::LoadingStrictnessLevel, std::function<void (String const&)>) @ 0x000000001cbafb45
9. /build/src/Storages/StorageMergeTree.cpp:104: DB::StorageMergeTree::StorageMergeTree(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata const&, DB::LoadingStrictnessLevel, std::shared_ptr<DB::Context>, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>) @ 0x000000001d0eb7b6
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageMergeTree* std::construct_at[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::StorageMergeTree*>(DB::StorageMergeTree*, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eb3b7
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageMergeTree>>::construct[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void, void>(std::allocator<DB::StorageMergeTree>&, DB::StorageMergeTree*, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eb0c7
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageMergeTree, std::allocator<DB::StorageMergeTree>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>>(std::allocator<DB::StorageMergeTree>, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0ead75
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageMergeTree> std::allocate_shared[abi:v15000]<DB::StorageMergeTree, std::allocator<DB::StorageMergeTree>, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void>(std::allocator<DB::StorageMergeTree> const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eaba7
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageMergeTree> std::make_shared[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void>(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0e7d6d
15. /build/src/Storages/MergeTree/registerStorageMergeTree.cpp:741: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000001d0e58aa
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000193086fd
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000193086bd
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001930867d
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019308645
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
22. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
23. /build/src/Interpreters/InterpreterCreateQuery.cpp:1493: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x000000001b01ec0c
24. /build/src/Interpreters/InterpreterCreateQuery.cpp:1313: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b018b66
25. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
26. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
27. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
28. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
29. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
30. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
31. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68

2024.11.05 19:56:52.902647 [ 3800 ] {d6c2e04e-96ee-4077-bb07-49d3a2f76d5f} <Error> TCPHandler: Code: 169. DB::Exception: is_deleted column (is_deleted) for storage ReplacingMergeTree must have type UInt8. Provided column of type String. (BAD_TYPE_OF_FIELD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String const&, String&&) @ 0x000000001a16d768
6. /build/src/Storages/MergeTree/MergeTreeData.cpp:968: DB::MergeTreeData::MergingParams::check(DB::StorageInMemoryMetadata const&) const::$_2::operator()(bool, String const&) const @ 0x000000001cbb8606
7. /build/src/Storages/MergeTree/MergeTreeData.cpp:1017: DB::MergeTreeData::MergingParams::check(DB::StorageInMemoryMetadata const&) const @ 0x000000001cbb1436
8. /build/src/Storages/MergeTree/MergeTreeData.cpp:401: DB::MergeTreeData::MergeTreeData(DB::StorageID const&, DB::StorageInMemoryMetadata const&, std::shared_ptr<DB::Context>, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, bool, DB::LoadingStrictnessLevel, std::function<void (String const&)>) @ 0x000000001cbafb45
9. /build/src/Storages/StorageMergeTree.cpp:104: DB::StorageMergeTree::StorageMergeTree(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata const&, DB::LoadingStrictnessLevel, std::shared_ptr<DB::Context>, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>) @ 0x000000001d0eb7b6
10. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageMergeTree* std::construct_at[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, DB::StorageMergeTree*>(DB::StorageMergeTree*, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eb3b7
11. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageMergeTree>>::construct[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void, void>(std::allocator<DB::StorageMergeTree>&, DB::StorageMergeTree*, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eb0c7
12. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageMergeTree, std::allocator<DB::StorageMergeTree>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>>(std::allocator<DB::StorageMergeTree>, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0ead75
13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageMergeTree> std::allocate_shared[abi:v15000]<DB::StorageMergeTree, std::allocator<DB::StorageMergeTree>, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void>(std::allocator<DB::StorageMergeTree> const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0eaba7
14. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageMergeTree> std::make_shared[abi:v15000]<DB::StorageMergeTree, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, void>(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, DB::LoadingStrictnessLevel const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&) @ 0x000000001d0e7d6d
15. /build/src/Storages/MergeTree/registerStorageMergeTree.cpp:741: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000001d0e58aa
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000193086fd
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000193086bd
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001930867d
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000019308645
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
22. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
23. /build/src/Interpreters/InterpreterCreateQuery.cpp:1493: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x000000001b01ec0c
24. /build/src/Interpreters/InterpreterCreateQuery.cpp:1313: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000001b018b66
25. /build/src/Interpreters/InterpreterCreateQuery.cpp:0: DB::InterpreterCreateQuery::execute() @ 0x000000001b021896
26. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
27. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
28. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
29. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
30. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
31. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68

2024.11.05 19:56:53.580340 [ 5794 ] {ffb4e18b-4f35-4123-8d24-e02a050acfd5} <Error> executeQuery: Code: 62. DB::Exception: Syntax error: failed at position 8 ('('): ([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([. Unmatched parentheses: ([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([. (SYNTAX_ERROR) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35310) (comment: 00984_parser_stack_overflow.sh) (in query: SELECT ([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:89: DB::Exception::Exception(String const&, int, bool) @ 0x000000000a9434aa
5. /build/src/Common/Exception.h:97: DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000a943455
6. /build/src/Parsers/parseQuery.cpp:373: DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x000000001e4065a1
7. /build/src/Parsers/parseQuery.cpp:386: DB::parseQuery(DB::IParser&, char const*, char const*, String const&, unsigned long, unsigned long, unsigned long) @ 0x000000001e406692
8. /build/src/Interpreters/executeQuery.cpp:760: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbae505
9. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
10. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
11. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
12. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
13. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
14. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
15. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
16. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
17. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
18. ? @ 0x00007f3c91188ac3
19. ? @ 0x00007f3c9121a850

2024.11.05 19:56:53.602198 [ 5794 ] {ffb4e18b-4f35-4123-8d24-e02a050acfd5} <Error> DynamicQueryHandler: Code: 62. DB::Exception: Syntax error: failed at position 8 ('('): ([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([. Unmatched parentheses: ([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([([. (SYNTAX_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:89: DB::Exception::Exception(String const&, int, bool) @ 0x000000000a9434aa
5. /build/src/Common/Exception.h:97: DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000a943455
6. /build/src/Parsers/parseQuery.cpp:373: DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x000000001e4065a1
7. /build/src/Parsers/parseQuery.cpp:386: DB::parseQuery(DB::IParser&, char const*, char const*, String const&, unsigned long, unsigned long, unsigned long) @ 0x000000001e406692
8. /build/src/Interpreters/executeQuery.cpp:760: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbae505
9. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
10. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
11. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
12. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
13. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
14. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
15. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
16. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
17. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
18. ? @ 0x00007f3c91188ac3
19. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:53.678074 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:54.637366 [ 3800 ] {60758d7d-88e5-4e05-b910-f8c3d7337e7d} <Error> executeQuery: Code: 344. DB::Exception: Experimental merges with CLEANUP are not allowed. (SUPPORT_IS_DISABLED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52026) (comment: 02490_replacing_merge_tree_is_deleted_column.sql) (in query: OPTIMIZE TABLE test FINAL CLEANUP;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/StorageMergeTree.cpp:1577: DB::StorageMergeTree::optimize(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::IAST> const&, bool, bool, std::vector<String, std::allocator<String>> const&, bool, std::shared_ptr<DB::Context const>) @ 0x000000001d101ccd
7. /build/src/Interpreters/InterpreterOptimizeQuery.cpp:84: DB::InterpreterOptimizeQuery::execute() @ 0x000000001bc1c745
8. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
9. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
10. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:56:54.649476 [ 3800 ] {60758d7d-88e5-4e05-b910-f8c3d7337e7d} <Error> TCPHandler: Code: 344. DB::Exception: Experimental merges with CLEANUP are not allowed. (SUPPORT_IS_DISABLED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/StorageMergeTree.cpp:1577: DB::StorageMergeTree::optimize(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::IAST> const&, bool, bool, std::vector<String, std::allocator<String>> const&, bool, std::shared_ptr<DB::Context const>) @ 0x000000001d101ccd
7. /build/src/Interpreters/InterpreterOptimizeQuery.cpp:84: DB::InterpreterOptimizeQuery::execute() @ 0x000000001bc1c745
8. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
9. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
10. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:56:55.372581 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:57.993760 [ 5793 ] {a3e09062-05f2-4cc4-ab3a-2e47f9ddaf9d} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=32): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:48336) (comment: 00042_any_left_join.sql) (in query: SELECT EventDate, hits, visits FROM ( SELECT EventDate, count() AS hits FROM test.hits GROUP BY EventDate ) ANY LEFT JOIN ( SELECT StartDate AS EventDate, sum(Sign) AS visits FROM test.visits GROUP BY EventDate ) USING EventDate ORDER BY hits DESC LIMIT 10 SETTINGS joined_subquery_requires_alias = 0;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:56:58.090761 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:58.458133 [ 5215 ] {2d7bc787-e1a5-41a5-9c57-7bcc6f482549} <Error> executeQuery: Code: 167. DB::Exception: AST is too deep. Maximum: 1000. (TOO_DEEP_AST) (version 24.3.12.76.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43744) (comment: 00984_parser_stack_overflow.sh) (in query: SELECT 1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long&>::type>, unsigned long&) @ 0x000000000b4610f3
6. /build/src/Parsers/IAST.cpp:157: DB::IAST::checkDepthImpl(unsigned long) const @ 0x000000001e381ea2
7. /build/src/Parsers/IAST.h:98: DB::IAST::checkDepth(unsigned long) const @ 0x000000001bbc0d3d
8. /build/src/Parsers/parseQuery.cpp:271: DB::tryParseQuery(DB::IParser&, char const*&, char const*, String&, bool, String const&, bool, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001e405188
9. /build/src/Parsers/parseQuery.cpp:366: DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x000000001e406511
10. /build/src/Parsers/parseQuery.cpp:386: DB::parseQuery(DB::IParser&, char const*, char const*, String const&, unsigned long, unsigned long, unsigned long) @ 0x000000001e406692
11. /build/src/Interpreters/executeQuery.cpp:760: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbae505
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:56:58.478318 [ 5215 ] {2d7bc787-e1a5-41a5-9c57-7bcc6f482549} <Error> DynamicQueryHandler: Code: 167. DB::Exception: AST is too deep. Maximum: 1000. (TOO_DEEP_AST), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<unsigned long&>(int, FormatStringHelperImpl<std::type_identity<unsigned long&>::type>, unsigned long&) @ 0x000000000b4610f3
6. /build/src/Parsers/IAST.cpp:157: DB::IAST::checkDepthImpl(unsigned long) const @ 0x000000001e381ea2
7. /build/src/Parsers/IAST.h:98: DB::IAST::checkDepth(unsigned long) const @ 0x000000001bbc0d3d
8. /build/src/Parsers/parseQuery.cpp:271: DB::tryParseQuery(DB::IParser&, char const*&, char const*, String&, bool, String const&, bool, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001e405188
9. /build/src/Parsers/parseQuery.cpp:366: DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x000000001e406511
10. /build/src/Parsers/parseQuery.cpp:386: DB::parseQuery(DB::IParser&, char const*, char const*, String const&, unsigned long, unsigned long, unsigned long) @ 0x000000001e406692
11. /build/src/Interpreters/executeQuery.cpp:760: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbae505
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:56:59.026166 [ 3800 ] {f96bf58c-1d54-4772-bc9b-891ceefc0697} <Error> executeQuery: Code: 344. DB::Exception: Experimental merges with CLEANUP are not allowed. (SUPPORT_IS_DISABLED) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52026) (comment: 02490_replacing_merge_tree_is_deleted_column.sql) (in query: OPTIMIZE TABLE test FINAL CLEANUP;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/StorageReplicatedMergeTree.cpp:5747: DB::StorageReplicatedMergeTree::optimize(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::IAST> const&, bool, bool, std::vector<String, std::allocator<String>> const&, bool, std::shared_ptr<DB::Context const>) @ 0x000000001c54cd3c
7. /build/src/Interpreters/InterpreterOptimizeQuery.cpp:84: DB::InterpreterOptimizeQuery::execute() @ 0x000000001bc1c745
8. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
9. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
10. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:56:59.035980 [ 3800 ] {f96bf58c-1d54-4772-bc9b-891ceefc0697} <Error> TCPHandler: Code: 344. DB::Exception: Experimental merges with CLEANUP are not allowed. (SUPPORT_IS_DISABLED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/StorageReplicatedMergeTree.cpp:5747: DB::StorageReplicatedMergeTree::optimize(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::IAST> const&, bool, bool, std::vector<String, std::allocator<String>> const&, bool, std::shared_ptr<DB::Context const>) @ 0x000000001c54cd3c
7. /build/src/Interpreters/InterpreterOptimizeQuery.cpp:84: DB::InterpreterOptimizeQuery::execute() @ 0x000000001bc1c745
8. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
9. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
10. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:57:00.609465 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:01.221042 [ 5793 ] {a3e09062-05f2-4cc4-ab3a-2e47f9ddaf9d} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=32): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:57:02.130798 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:03.367988 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:03.639893 [ 3800 ] {c2f4deb3-4b12-466f-a334-e5e98e97d72a} <Error> executeQuery: Code: 388. DB::Exception: Cannot OPTIMIZE with CLEANUP table: only ReplacingMergeTree can be CLEANUP. (CANNOT_ASSIGN_OPTIMIZE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52026) (comment: 02490_replacing_merge_tree_is_deleted_column.sql) (in query: OPTIMIZE TABLE testMT FINAL CLEANUP;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Storages/StorageMergeTree.cpp:1573: DB::StorageMergeTree::optimize(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::IAST> const&, bool, bool, std::vector<String, std::allocator<String>> const&, bool, std::shared_ptr<DB::Context const>) @ 0x000000001d101b34
7. /build/src/Interpreters/InterpreterOptimizeQuery.cpp:84: DB::InterpreterOptimizeQuery::execute() @ 0x000000001bc1c745
8. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
9. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
10. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:57:03.646497 [ 3800 ] {c2f4deb3-4b12-466f-a334-e5e98e97d72a} <Error> TCPHandler: Code: 388. DB::Exception: Cannot OPTIMIZE with CLEANUP table: only ReplacingMergeTree can be CLEANUP. (CANNOT_ASSIGN_OPTIMIZE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Storages/StorageMergeTree.cpp:1573: DB::StorageMergeTree::optimize(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::IAST> const&, bool, bool, std::vector<String, std::allocator<String>> const&, bool, std::shared_ptr<DB::Context const>) @ 0x000000001d101b34
7. /build/src/Interpreters/InterpreterOptimizeQuery.cpp:84: DB::InterpreterOptimizeQuery::execute() @ 0x000000001bc1c745
8. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
9. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
10. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:57:04.396444 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:06.066573 [ 5793 ] {95c2c980-7806-4fd2-b426-9037b58d5eef} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:40300) (comment: 00066_sorting_distributed_many_replicas.sql) (in query: SELECT EventTime::DateTime('Asia/Dubai') FROM remote('127.0.0.{1|2}', test, hits) ORDER BY EventTime DESC LIMIT 10 ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:57:06.239662 [ 5793 ] {95c2c980-7806-4fd2-b426-9037b58d5eef} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:57:06.277054 [ 3800 ] {37465d1c-504c-40e2-8533-3bb0dcb8231d} <Error> executeQuery: Code: 388. DB::Exception: Cannot OPTIMIZE with CLEANUP table: only ReplacingMergeTree can be CLEANUP. (CANNOT_ASSIGN_OPTIMIZE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52026) (comment: 02490_replacing_merge_tree_is_deleted_column.sql) (in query: OPTIMIZE TABLE testSummingMT FINAL CLEANUP;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Storages/StorageMergeTree.cpp:1573: DB::StorageMergeTree::optimize(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::IAST> const&, bool, bool, std::vector<String, std::allocator<String>> const&, bool, std::shared_ptr<DB::Context const>) @ 0x000000001d101b34
7. /build/src/Interpreters/InterpreterOptimizeQuery.cpp:84: DB::InterpreterOptimizeQuery::execute() @ 0x000000001bc1c745
8. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
9. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
10. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:57:06.289865 [ 3800 ] {37465d1c-504c-40e2-8533-3bb0dcb8231d} <Error> TCPHandler: Code: 388. DB::Exception: Cannot OPTIMIZE with CLEANUP table: only ReplacingMergeTree can be CLEANUP. (CANNOT_ASSIGN_OPTIMIZE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Storages/StorageMergeTree.cpp:1573: DB::StorageMergeTree::optimize(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::IAST> const&, bool, bool, std::vector<String, std::allocator<String>> const&, bool, std::shared_ptr<DB::Context const>) @ 0x000000001d101b34
7. /build/src/Interpreters/InterpreterOptimizeQuery.cpp:84: DB::InterpreterOptimizeQuery::execute() @ 0x000000001bc1c745
8. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
9. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
10. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:57:08.441012 [ 3800 ] {37c185fa-95e4-4723-9874-b12159bff93d} <Error> executeQuery: Code: 388. DB::Exception: Cannot OPTIMIZE with CLEANUP table: only ReplacingMergeTree can be CLEANUP. (CANNOT_ASSIGN_OPTIMIZE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52026) (comment: 02490_replacing_merge_tree_is_deleted_column.sql) (in query: OPTIMIZE TABLE testAggregatingMT FINAL CLEANUP;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Storages/StorageMergeTree.cpp:1573: DB::StorageMergeTree::optimize(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::IAST> const&, bool, bool, std::vector<String, std::allocator<String>> const&, bool, std::shared_ptr<DB::Context const>) @ 0x000000001d101b34
7. /build/src/Interpreters/InterpreterOptimizeQuery.cpp:84: DB::InterpreterOptimizeQuery::execute() @ 0x000000001bc1c745
8. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
9. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
10. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:57:08.450035 [ 3800 ] {37c185fa-95e4-4723-9874-b12159bff93d} <Error> TCPHandler: Code: 388. DB::Exception: Cannot OPTIMIZE with CLEANUP table: only ReplacingMergeTree can be CLEANUP. (CANNOT_ASSIGN_OPTIMIZE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Storages/StorageMergeTree.cpp:1573: DB::StorageMergeTree::optimize(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::IAST> const&, bool, bool, std::vector<String, std::allocator<String>> const&, bool, std::shared_ptr<DB::Context const>) @ 0x000000001d101b34
7. /build/src/Interpreters/InterpreterOptimizeQuery.cpp:84: DB::InterpreterOptimizeQuery::execute() @ 0x000000001bc1c745
8. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
9. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
10. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:57:08.667248 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/eqj/bdtaairfpvzdevaqqsrlxqmhnaihh, total size 285212672, count 285212672, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 724c069b-732f-48f0-b2ff-f40488eae99b, upload has finished false.
2024.11.05 19:57:08.667933 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/eqj/bdtaairfpvzdevaqqsrlxqmhnaihh, total size 285212672, count 285212672, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 724c069b-732f-48f0-b2ff-f40488eae99b, upload has finished false
2024.11.05 19:57:08.775406 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/eqj/bdtaairfpvzdevaqqsrlxqmhnaihh, total size 285212672, count 285212672, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 724c069b-732f-48f0-b2ff-f40488eae99b, upload has finished false
2024.11.05 19:57:08.827145 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 59 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:11.662555 [ 3800 ] {3943320f-df70-4ce9-a404-b1e37c08c2dc} <Error> executeQuery: Code: 388. DB::Exception: Cannot OPTIMIZE with CLEANUP table: only ReplacingMergeTree can be CLEANUP. (CANNOT_ASSIGN_OPTIMIZE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52026) (comment: 02490_replacing_merge_tree_is_deleted_column.sql) (in query: OPTIMIZE TABLE testCollapsingMT FINAL CLEANUP;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Storages/StorageMergeTree.cpp:1573: DB::StorageMergeTree::optimize(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::IAST> const&, bool, bool, std::vector<String, std::allocator<String>> const&, bool, std::shared_ptr<DB::Context const>) @ 0x000000001d101b34
7. /build/src/Interpreters/InterpreterOptimizeQuery.cpp:84: DB::InterpreterOptimizeQuery::execute() @ 0x000000001bc1c745
8. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
9. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
10. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:57:11.676380 [ 3800 ] {3943320f-df70-4ce9-a404-b1e37c08c2dc} <Error> TCPHandler: Code: 388. DB::Exception: Cannot OPTIMIZE with CLEANUP table: only ReplacingMergeTree can be CLEANUP. (CANNOT_ASSIGN_OPTIMIZE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Storages/StorageMergeTree.cpp:1573: DB::StorageMergeTree::optimize(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::IAST> const&, bool, bool, std::vector<String, std::allocator<String>> const&, bool, std::shared_ptr<DB::Context const>) @ 0x000000001d101b34
7. /build/src/Interpreters/InterpreterOptimizeQuery.cpp:84: DB::InterpreterOptimizeQuery::execute() @ 0x000000001bc1c745
8. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
9. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
10. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:57:13.327511 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:14.377400 [ 3800 ] {6c60f68f-07c0-46a6-96bc-9aad65fcac9e} <Error> executeQuery: Code: 388. DB::Exception: Cannot OPTIMIZE with CLEANUP table: only ReplacingMergeTree can be CLEANUP. (CANNOT_ASSIGN_OPTIMIZE) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52026) (comment: 02490_replacing_merge_tree_is_deleted_column.sql) (in query: OPTIMIZE TABLE testVersionedCMT FINAL CLEANUP;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Storages/StorageMergeTree.cpp:1573: DB::StorageMergeTree::optimize(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::IAST> const&, bool, bool, std::vector<String, std::allocator<String>> const&, bool, std::shared_ptr<DB::Context const>) @ 0x000000001d101b34
7. /build/src/Interpreters/InterpreterOptimizeQuery.cpp:84: DB::InterpreterOptimizeQuery::execute() @ 0x000000001bc1c745
8. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
9. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
10. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:57:14.445390 [ 3800 ] {6c60f68f-07c0-46a6-96bc-9aad65fcac9e} <Error> TCPHandler: Code: 388. DB::Exception: Cannot OPTIMIZE with CLEANUP table: only ReplacingMergeTree can be CLEANUP. (CANNOT_ASSIGN_OPTIMIZE), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000a942f53
6. /build/src/Storages/StorageMergeTree.cpp:1573: DB::StorageMergeTree::optimize(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::IAST> const&, bool, bool, std::vector<String, std::allocator<String>> const&, bool, std::shared_ptr<DB::Context const>) @ 0x000000001d101b34
7. /build/src/Interpreters/InterpreterOptimizeQuery.cpp:84: DB::InterpreterOptimizeQuery::execute() @ 0x000000001bc1c745
8. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
9. /build/src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001bbacfaa
10. /build/src/Server/TCPHandler.cpp:517: DB::TCPHandler::runImpl() @ 0x000000001d470660
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850

2024.11.05 19:57:23.460460 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column IsNew): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_6_6_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 0 with max_rows_to_read = 6868): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:25.424544 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:25.590882 [ 8198 ] {a820dcb9-075b-4ffe-9c1a-4757b266c460} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:40340) (comment: 00004_top_counters.sql) (in query: SELECT CounterID, count() AS c FROM test.hits GROUP BY CounterID ORDER BY c DESC LIMIT 10;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:57:28.983690 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_121_21}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=7). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:45: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce610d
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
24. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
25. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
26. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
27. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
28. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:29.826889 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/jeb/igprnqervefkmxbqoathhfiemeqga, total size 234881024, count 234881024, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 87c1bcbc-1d25-4dca-8757-5e0fa21e2c1b, upload has finished false.
2024.11.05 19:57:29.832874 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/jeb/igprnqervefkmxbqoathhfiemeqga, total size 234881024, count 234881024, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 87c1bcbc-1d25-4dca-8757-5e0fa21e2c1b, upload has finished false
2024.11.05 19:57:29.941210 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/jeb/igprnqervefkmxbqoathhfiemeqga, total size 234881024, count 234881024, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 87c1bcbc-1d25-4dca-8757-5e0fa21e2c1b, upload has finished false
2024.11.05 19:57:29.992483 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 52 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:31.980593 [ 8198 ] {a820dcb9-075b-4ffe-9c1a-4757b266c460} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:57:34.636982 [ 5215 ] {642a9354-3997-4a0b-a296-578e0881f56b} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:36258) (comment: 00157_cache_dictionary.sql) (in query: INSERT INTO test.hits_1m SELECT * FROM test.hits LIMIT 1000000;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:57:34.786401 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0a233324-63e4-4351-8154-d23ef4927e18::202411_1_116_23}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:34.895826 [ 5215 ] {642a9354-3997-4a0b-a296-578e0881f56b} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:57:36.148804 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/eok/hxauvyxpdkfwxhcvlbmtajdassioj, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6680af92-dcc2-4486-8882-f92ad4e1cc3a, upload has finished false.
2024.11.05 19:57:36.151448 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/eok/hxauvyxpdkfwxhcvlbmtajdassioj, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6680af92-dcc2-4486-8882-f92ad4e1cc3a, upload has finished false
2024.11.05 19:57:36.179087 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/eok/hxauvyxpdkfwxhcvlbmtajdassioj, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6680af92-dcc2-4486-8882-f92ad4e1cc3a, upload has finished false
2024.11.05 19:57:36.238585 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 15 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:38.827322 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:39.307243 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {c04f5a1b-d832-4de4-96b0-dec4e591ea83::202411_1_115_23}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:39.427355 [ 3112 ] {} <Error> void DB::StorageWindowView::threadFuncFireEvent(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:42.137193 [ 3641 ] {} <Error> void DB::SystemLog<DB::PartLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::PartLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:42.567339 [ 5215 ] {a897d083-ff3e-48ae-be2b-6237c04c3755} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:36274) (comment: 00152_insert_different_granularity.sql) (in query: INSERT INTO fixed_granularity_table SELECT * FROM test.hits LIMIT 10;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:57:42.752449 [ 5215 ] {a897d083-ff3e-48ae-be2b-6237c04c3755} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:57:42.838079 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:42.855828 [ 8198 ] {cdb2d578-18d6-4de4-a131-2b315ba5bcc2} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=10, jobs=10). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:36280) (comment: 00088_global_in_one_shard_and_rows_before_limit.sql) (in query: SELECT EventDate, count() FROM remote('127.0.0.1', test.hits) WHERE UserID GLOBAL IN (SELECT UserID FROM test.hits) GROUP BY EventDate ORDER BY EventDate LIMIT 5 FORMAT JSONCompact;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:57:42.902402 [ 8198 ] {cdb2d578-18d6-4de4-a131-2b315ba5bcc2} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=10, jobs=10). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1

2024.11.05 19:57:43.198579 [ 5793 ] {5c10d0c9-c92b-4f6c-a256-04da2654b9e6} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:33716) (comment: 01058_window_view_event_hop_to_strict_asc.sh) (in query: INSERT INTO mt VALUES ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:43: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce60f4
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375

2024.11.05 19:57:43.219061 [ 5793 ] {5c10d0c9-c92b-4f6c-a256-04da2654b9e6} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:43: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce60f4
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375

2024.11.05 19:57:43.479553 [ 3227 ] {} <Warning> test_3.rep_data (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.11.05 19:57:43.481167 [ 3220 ] {} <Warning> test_15.testSettingsR1 (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.11.05 19:57:43.482346 [ 3229 ] {} <Warning> test_15.testCleanupR1 (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.11.05 19:57:43.483811 [ 3230 ] {} <Warning> test_1.r_prop_table1 (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.11.05 19:57:43.485252 [ 3221 ] {} <Warning> test_1.r_prop_table2 (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.11.05 19:57:43.962084 [ 5795 ] {25e32113-455c-4738-a90f-b5d87bebfb6f} <Error> executeQuery: Code: 999. Coordination::Exception: Session expired (fault injected on send). (KEEPER_EXCEPTION) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46920) (comment: 00152_insert_different_granularity.sql) (in query: DROP DATABASE IF EXISTS test_k6cbek3q ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/ZooKeeper/IKeeper.h:475: Coordination::Exception::Exception<char const (&) [41]>(char const (&) [41], Coordination::Error) @ 0x000000001d0b1a04
6. /build/src/Common/ZooKeeper/IKeeper.h:509: Coordination::Exception Coordination::Exception::fromMessage<char const (&) [41]>(Coordination::Error, char const (&) [41]) @ 0x000000001d0b19a6
7. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:1605: Coordination::ZooKeeper::maybeInjectSendFault() @ 0x000000001e249d20
8. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:1183: Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x000000001e24993b
9. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:1400: Coordination::ZooKeeper::list(String const&, Coordination::ListRequestType, std::function<void (Coordination::ListResponse const&)>, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>) @ 0x000000001e24bb9d
10. /build/src/Common/ZooKeeper/ZooKeeper.cpp:1192: zkutil::ZooKeeper::asyncTryGetChildrenNoThrow(String const&, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>, Coordination::ListRequestType) @ 0x000000001e1b0afc
11. /build/src/Common/ZooKeeper/ZooKeeper.cpp:216: zkutil::ZooKeeper::getChildrenImpl(String const&, std::vector<String, std::allocator<String>>&, Coordination::Stat*, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>, Coordination::ListRequestType) @ 0x000000001e1b0694
12. /build/src/Common/ZooKeeper/ZooKeeper.cpp:289: zkutil::ZooKeeper::tryGetChildrenWatch(String const&, std::vector<String, std::allocator<String>>&, Coordination::Stat*, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>, Coordination::ListRequestType) @ 0x000000001e1b125b
13. /build/src/Common/ZooKeeper/ZooKeeper.cpp:265: zkutil::ZooKeeper::tryGetChildren(String const&, std::vector<String, std::allocator<String>>&, Coordination::Stat*, std::shared_ptr<Poco::Event> const&, Coordination::ListRequestType) @ 0x000000001e1b0e42
14. /build/src/Common/ZooKeeper/ZooKeeper.cpp:775: zkutil::ZooKeeper::tryRemoveChildrenRecursive(String const&, bool, zkutil::RemoveException) @ 0x000000001e1b6b5b
15. /build/src/Common/ZooKeeper/ZooKeeper.cpp:854: zkutil::ZooKeeper::tryRemoveRecursive(String const&) @ 0x000000001e1b75d2
16. /build/src/Databases/DatabaseReplicated.cpp:1314: DB::DatabaseReplicated::drop(std::shared_ptr<DB::Context const>) @ 0x000000001a1089f4
17. /build/src/Interpreters/DatabaseCatalog.cpp:607: DB::DatabaseCatalog::detachDatabase(std::shared_ptr<DB::Context const>, String const&, bool, bool) @ 0x000000001a672bb9
18. /build/src/Interpreters/InterpreterDropQuery.cpp:436: DB::InterpreterDropQuery::executeToDatabaseImpl(DB::ASTDropQuery const&, std::shared_ptr<DB::IDatabase>&, std::vector<StrongTypedef<wide::integer<128ul, unsigned int>, DB::UUIDTag>, std::allocator<StrongTypedef<wide::integer<128ul, unsigned int>, DB::UUIDTag>>>&) @ 0x000000001b03fcd3
19. /build/src/Interpreters/InterpreterDropQuery.cpp:335: DB::InterpreterDropQuery::executeToDatabase(DB::ASTDropQuery const&) @ 0x000000001b03bed9
20. /build/src/Interpreters/InterpreterDropQuery.cpp:81: DB::InterpreterDropQuery::execute() @ 0x000000001b03b604
21. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
22. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
23. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
24. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
25. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
26. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
27. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
28. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
29. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
30. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
31. ? @ 0x00007f3c91188ac3

2024.11.05 19:57:43.970853 [ 5795 ] {25e32113-455c-4738-a90f-b5d87bebfb6f} <Error> DynamicQueryHandler: Code: 999. Coordination::Exception: Session expired (fault injected on send). (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/ZooKeeper/IKeeper.h:475: Coordination::Exception::Exception<char const (&) [41]>(char const (&) [41], Coordination::Error) @ 0x000000001d0b1a04
6. /build/src/Common/ZooKeeper/IKeeper.h:509: Coordination::Exception Coordination::Exception::fromMessage<char const (&) [41]>(Coordination::Error, char const (&) [41]) @ 0x000000001d0b19a6
7. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:1605: Coordination::ZooKeeper::maybeInjectSendFault() @ 0x000000001e249d20
8. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:1183: Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x000000001e24993b
9. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:1400: Coordination::ZooKeeper::list(String const&, Coordination::ListRequestType, std::function<void (Coordination::ListResponse const&)>, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>) @ 0x000000001e24bb9d
10. /build/src/Common/ZooKeeper/ZooKeeper.cpp:1192: zkutil::ZooKeeper::asyncTryGetChildrenNoThrow(String const&, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>, Coordination::ListRequestType) @ 0x000000001e1b0afc
11. /build/src/Common/ZooKeeper/ZooKeeper.cpp:216: zkutil::ZooKeeper::getChildrenImpl(String const&, std::vector<String, std::allocator<String>>&, Coordination::Stat*, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>, Coordination::ListRequestType) @ 0x000000001e1b0694
12. /build/src/Common/ZooKeeper/ZooKeeper.cpp:289: zkutil::ZooKeeper::tryGetChildrenWatch(String const&, std::vector<String, std::allocator<String>>&, Coordination::Stat*, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>, Coordination::ListRequestType) @ 0x000000001e1b125b
13. /build/src/Common/ZooKeeper/ZooKeeper.cpp:265: zkutil::ZooKeeper::tryGetChildren(String const&, std::vector<String, std::allocator<String>>&, Coordination::Stat*, std::shared_ptr<Poco::Event> const&, Coordination::ListRequestType) @ 0x000000001e1b0e42
14. /build/src/Common/ZooKeeper/ZooKeeper.cpp:775: zkutil::ZooKeeper::tryRemoveChildrenRecursive(String const&, bool, zkutil::RemoveException) @ 0x000000001e1b6b5b
15. /build/src/Common/ZooKeeper/ZooKeeper.cpp:854: zkutil::ZooKeeper::tryRemoveRecursive(String const&) @ 0x000000001e1b75d2
16. /build/src/Databases/DatabaseReplicated.cpp:1314: DB::DatabaseReplicated::drop(std::shared_ptr<DB::Context const>) @ 0x000000001a1089f4
17. /build/src/Interpreters/DatabaseCatalog.cpp:607: DB::DatabaseCatalog::detachDatabase(std::shared_ptr<DB::Context const>, String const&, bool, bool) @ 0x000000001a672bb9
18. /build/src/Interpreters/InterpreterDropQuery.cpp:436: DB::InterpreterDropQuery::executeToDatabaseImpl(DB::ASTDropQuery const&, std::shared_ptr<DB::IDatabase>&, std::vector<StrongTypedef<wide::integer<128ul, unsigned int>, DB::UUIDTag>, std::allocator<StrongTypedef<wide::integer<128ul, unsigned int>, DB::UUIDTag>>>&) @ 0x000000001b03fcd3
19. /build/src/Interpreters/InterpreterDropQuery.cpp:335: DB::InterpreterDropQuery::executeToDatabase(DB::ASTDropQuery const&) @ 0x000000001b03bed9
20. /build/src/Interpreters/InterpreterDropQuery.cpp:81: DB::InterpreterDropQuery::execute() @ 0x000000001b03b604
21. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
22. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
23. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
24. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
25. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
26. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
27. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
28. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
29. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
30. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
31. ? @ 0x00007f3c91188ac3
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:44.433437 [ 3800 ] {3d03eefc-4912-4fca-9aeb-5872a80021ee} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:1ca6e31c-1352-4d47-b705-4de68753183c'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46936) (comment: 00154_avro.sql) (in query: CREATE DATABASE IF NOT EXISTS test_opk4h6dc ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:44.446017 [ 3800 ] {3d03eefc-4912-4fca-9aeb-5872a80021ee} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:1ca6e31c-1352-4d47-b705-4de68753183c'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:44.445946 [ 3205 ] {} <Error> void DB::StorageWindowView::threadFuncFireEvent(): Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=14, jobs=14). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:263: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7a1e
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:306: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001d516831
11. /build/src/Processors/Executors/PipelineExecutor.cpp:239: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001d516a37
12. /build/src/Processors/Executors/PipelineExecutor.cpp:373: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001d5179d8
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d517935
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d517915
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d5178f5
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d5178c0
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:44.731411 [ 5795 ] {f4231a75-e49b-4677-ab4f-9bfad435a2cf} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:9ab93853-4e39-4cf6-9d32-3e441bf0f8fa'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46950) (comment: 00030_array_enumerate_uniq.sql) (in query: CREATE DATABASE IF NOT EXISTS test_w9ucik7b ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:44.738262 [ 5795 ] {f4231a75-e49b-4677-ab4f-9bfad435a2cf} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:9ab93853-4e39-4cf6-9d32-3e441bf0f8fa'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:44.877762 [ 3800 ] {07e23282-0cd3-4728-a2a6-b3bfbb39ffaf} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:62d22821-533a-42da-9b76-2bae37259de1'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46964) (comment: 00079_array_join_not_used_joined_column.sql) (in query: CREATE DATABASE IF NOT EXISTS test_3gniyhsu ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:44.883158 [ 3800 ] {07e23282-0cd3-4728-a2a6-b3bfbb39ffaf} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:62d22821-533a-42da-9b76-2bae37259de1'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:45.038832 [ 5795 ] {7abd9a9e-2e8b-4b78-b8f5-899de259d6e5} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:c77f87a5-6c6f-431b-8d24-7af51e744c35'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46974) (comment: 00016_any_if_distributed_cond_always_false.sql) (in query: CREATE DATABASE IF NOT EXISTS test_3q3u73mk ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:45.060614 [ 5795 ] {7abd9a9e-2e8b-4b78-b8f5-899de259d6e5} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:c77f87a5-6c6f-431b-8d24-7af51e744c35'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:45.205438 [ 3800 ] {7213d569-b1ef-4101-9b08-ce11fbad89bd} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:dfe5563a-5936-468d-82f9-10dbbacda2ce'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46984) (comment: 00033_aggregate_key_string.sql) (in query: CREATE DATABASE IF NOT EXISTS test_liatufrw ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:45.215000 [ 3800 ] {7213d569-b1ef-4101-9b08-ce11fbad89bd} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:dfe5563a-5936-468d-82f9-10dbbacda2ce'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:45.443853 [ 5795 ] {83542b14-4e70-46a5-86d1-7080707e5172} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:578057db-7eaf-49f6-a93e-1b7145dee6a8'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:46990) (comment: 00039_primary_key.sql) (in query: CREATE DATABASE IF NOT EXISTS test_eclxpyxf ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:45.453835 [ 5795 ] {83542b14-4e70-46a5-86d1-7080707e5172} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:578057db-7eaf-49f6-a93e-1b7145dee6a8'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:45.672972 [ 3800 ] {391ec356-0d52-43eb-8b1e-f1411d374f33} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:ac2d1d83-4bd0-4aea-bdef-3b9ec772e2f1'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47000) (comment: 00047_bar.sql) (in query: CREATE DATABASE IF NOT EXISTS test_5d547kur ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:45.692667 [ 3800 ] {391ec356-0d52-43eb-8b1e-f1411d374f33} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:ac2d1d83-4bd0-4aea-bdef-3b9ec772e2f1'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:45.961791 [ 5795 ] {874a6a28-48ad-48bd-baac-4701b3c2b103} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:45b152f3-4118-4579-bb68-8ed69bbbff90'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47006) (comment: 00158_cache_dictionary_has.sql) (in query: CREATE DATABASE IF NOT EXISTS test_vwzj9ia1 ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:45.988270 [ 5795 ] {874a6a28-48ad-48bd-baac-4701b3c2b103} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:45b152f3-4118-4579-bb68-8ed69bbbff90'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:46.229713 [ 5215 ] {7d26d336-b79b-4a30-8290-2ed85964ebfd} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:10bdf32a-3839-4a34-acf4-58f4a5ac7280'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47034) (comment: 00051_min_max_array.sql) (in query: CREATE DATABASE IF NOT EXISTS test_44r0y3hl ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:46.233737 [ 5215 ] {7d26d336-b79b-4a30-8290-2ed85964ebfd} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:10bdf32a-3839-4a34-acf4-58f4a5ac7280'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:46.485433 [ 5795 ] {66ac825d-7a9d-43d9-99b6-3c7f6a656db8} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:25f16d85-0df5-49ce-870b-4ce785fd89a3'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47036) (comment: 00172_hits_joins.gen.sql) (in query: CREATE DATABASE IF NOT EXISTS test_5ulhexec ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:46.496019 [ 5795 ] {66ac825d-7a9d-43d9-99b6-3c7f6a656db8} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:25f16d85-0df5-49ce-870b-4ce785fd89a3'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:46.757788 [ 5215 ] {294a0315-5094-4e5b-9c16-c35e77fec92c} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:e23ae9b0-b34f-40af-95ae-b7c01409d68c'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47050) (comment: 00014_filtering_arrays.sql) (in query: CREATE DATABASE IF NOT EXISTS test_4a6tp628 ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:46.764487 [ 5215 ] {294a0315-5094-4e5b-9c16-c35e77fec92c} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:e23ae9b0-b34f-40af-95ae-b7c01409d68c'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:46.941353 [ 5795 ] {78669a07-149d-47f3-bfdd-1b5592aa0cfc} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:3be9b75c-f66b-454e-9904-2698e2b2f4ba'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47052) (comment: 00040_aggregating_materialized_view.sql) (in query: CREATE DATABASE IF NOT EXISTS test_yl994zjk ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:46.966381 [ 5795 ] {78669a07-149d-47f3-bfdd-1b5592aa0cfc} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:3be9b75c-f66b-454e-9904-2698e2b2f4ba'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:47.223780 [ 5215 ] {30ce0914-8369-4b75-9295-d4cd3abedece} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:5085676b-2801-4bab-bada-313638ed3ef9'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47054) (comment: 00031_array_enumerate_uniq.sql) (in query: CREATE DATABASE IF NOT EXISTS test_5o6gy75p ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:47.238619 [ 5215 ] {30ce0914-8369-4b75-9295-d4cd3abedece} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:5085676b-2801-4bab-bada-313638ed3ef9'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:47.458763 [ 3800 ] {f0b6af30-cd5a-44b2-a99d-c91f5ee3e6f8} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:df492e52-1820-43bb-97f2-799ac1da8c2e'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47062) (comment: 00011_sorting.sql) (in query: CREATE DATABASE IF NOT EXISTS test_fvfimqak ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:47.479601 [ 3800 ] {f0b6af30-cd5a-44b2-a99d-c91f5ee3e6f8} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:df492e52-1820-43bb-97f2-799ac1da8c2e'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:47.628021 [ 5215 ] {49e25fc5-b70b-488f-86c6-c5a4bcc0b50d} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:5a743474-c65f-4235-955b-90bfd2c15ea9'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47078) (comment: 00056_view.sql) (in query: CREATE DATABASE IF NOT EXISTS test_mipdlkp5 ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:47.642080 [ 5215 ] {49e25fc5-b70b-488f-86c6-c5a4bcc0b50d} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:5a743474-c65f-4235-955b-90bfd2c15ea9'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:48.615388 [ 5215 ] {6f9ed03a-8a31-4734-b8a2-f43f1089e277} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:9a87fdbb-f1df-41fa-8869-7b2ec187b6df'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47080) (comment: 02513_csv_bool_allow_crlf.sh) (in query: CREATE DATABASE IF NOT EXISTS test_bvcf7ujv ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:48.639304 [ 5215 ] {6f9ed03a-8a31-4734-b8a2-f43f1089e277} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:9a87fdbb-f1df-41fa-8869-7b2ec187b6df'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:48.890814 [ 5795 ] {b80a35f6-8f12-4f4a-8e9e-6169f67ace4b} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:6275d92a-befa-4cd3-8ece-855a417b63f9'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47088) (comment: 00754_first_significant_subdomain_more.sql) (in query: CREATE DATABASE IF NOT EXISTS test_5bw0zyl5 ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:48.940887 [ 5795 ] {b80a35f6-8f12-4f4a-8e9e-6169f67ace4b} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:6275d92a-befa-4cd3-8ece-855a417b63f9'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:49.121646 [ 5215 ] {c15eba18-3f7a-4e88-95b6-30b8bdbd817e} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:53015168-fb4a-4189-8f1f-57fd7c0e1d7a'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47092) (comment: 02560_with_fill_int256_int.sql) (in query: CREATE DATABASE IF NOT EXISTS test_6tetb2zm ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:49.133174 [ 5215 ] {c15eba18-3f7a-4e88-95b6-30b8bdbd817e} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:53015168-fb4a-4189-8f1f-57fd7c0e1d7a'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:49.331712 [ 5795 ] {f26b01d9-f214-42b3-ad11-73abf66f72b7} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:e0cd73d3-4883-4139-9215-bbb23475db1b'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47098) (comment: 00464_sort_all_constant_columns.sql) (in query: CREATE DATABASE IF NOT EXISTS test_neipnudu ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:49.406444 [ 5795 ] {f26b01d9-f214-42b3-ad11-73abf66f72b7} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:e0cd73d3-4883-4139-9215-bbb23475db1b'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:49.562708 [ 5215 ] {9185f92e-9578-4cde-bf5b-1e34760b2329} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:2154844b-73b1-47d1-b2e5-6c8db454db6d'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47112) (comment: 02669_alter_modify_to_nullable.sql) (in query: CREATE DATABASE IF NOT EXISTS test_qa0p8t15 ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:49.584919 [ 5215 ] {9185f92e-9578-4cde-bf5b-1e34760b2329} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:2154844b-73b1-47d1-b2e5-6c8db454db6d'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:49.740129 [ 5795 ] {7f230f67-2a33-4432-aa71-39c237db4655} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:c74bc53a-7285-48e6-9008-25167137e4c2'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47128) (comment: 02896_max_execution_time_with_break_overflow_mode.sql) (in query: CREATE DATABASE IF NOT EXISTS test_48o8n3ev ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:49.836485 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/xon/zplqvplikvftyrmgoaojdjhoxbdlr, total size 201326592, count 201326592, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id dbd22367-70d4-4a27-8b91-9af818824ce8, upload has finished false.
2024.11.05 19:57:49.836593 [ 5795 ] {7f230f67-2a33-4432-aa71-39c237db4655} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:c74bc53a-7285-48e6-9008-25167137e4c2'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:49.837605 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/xon/zplqvplikvftyrmgoaojdjhoxbdlr, total size 201326592, count 201326592, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id dbd22367-70d4-4a27-8b91-9af818824ce8, upload has finished false
2024.11.05 19:57:49.915194 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/xon/zplqvplikvftyrmgoaojdjhoxbdlr, total size 201326592, count 201326592, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id dbd22367-70d4-4a27-8b91-9af818824ce8, upload has finished false
2024.11.05 19:57:49.976991 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 42 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:50.040462 [ 5215 ] {fdbff71d-4e19-4113-b381-fbb45a9054a7} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:e9605240-dec7-4962-a861-5330437d0065'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47130) (comment: 02209_short_circuit_node_without_parents.sql) (in query: CREATE DATABASE IF NOT EXISTS test_ig4pzsfy ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:50.062365 [ 5215 ] {fdbff71d-4e19-4113-b381-fbb45a9054a7} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:e9605240-dec7-4962-a861-5330437d0065'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:50.220569 [ 5795 ] {72986011-9a11-44c9-a47e-4fbcaccbb5a7} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:934ff868-d3c2-4be5-851b-477ec2ba6e03'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47138) (comment: 01508_race_condition_rename_clear_zookeeper_long.sh) (in query: CREATE DATABASE IF NOT EXISTS test_ux3bo32u ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:50.248568 [ 5795 ] {72986011-9a11-44c9-a47e-4fbcaccbb5a7} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:934ff868-d3c2-4be5-851b-477ec2ba6e03'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:50.429190 [ 5215 ] {2cbb5ee0-e700-4f9f-ad8b-76af1609a0e4} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:728a7679-13e6-4a8e-bc5d-010395bb410a'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47144) (comment: 00464_array_element_out_of_range.sql) (in query: CREATE DATABASE IF NOT EXISTS test_9b6yebp7 ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:50.445790 [ 5215 ] {2cbb5ee0-e700-4f9f-ad8b-76af1609a0e4} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:728a7679-13e6-4a8e-bc5d-010395bb410a'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:50.645197 [ 5795 ] {abb86df7-e4d5-459c-96be-be76dbbe296d} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:8936d993-7eba-4696-9027-a06936eb84c7'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47152) (comment: 01572_kill_window_function.sh) (in query: CREATE DATABASE IF NOT EXISTS test_3gng3w88 ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:50.649004 [ 5795 ] {abb86df7-e4d5-459c-96be-be76dbbe296d} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:8936d993-7eba-4696-9027-a06936eb84c7'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:50.976140 [ 5215 ] {ef9f5c1b-e3ea-4c71-8540-3a7e964839c9} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:71bbc1ec-599f-45e5-920e-ffc038a7007c'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47158) (comment: 01710_projections.sql) (in query: CREATE DATABASE IF NOT EXISTS test_3xmx5pdn ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:51.005007 [ 5215 ] {ef9f5c1b-e3ea-4c71-8540-3a7e964839c9} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:71bbc1ec-599f-45e5-920e-ffc038a7007c'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:51.258425 [ 3800 ] {f5151741-af16-427f-92ee-886b07aba2ed} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:1f3caa26-8d11-435d-95ac-1983d27f3298'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47160) (comment: 02903_parameterized_view_explain_ast.sql) (in query: CREATE DATABASE IF NOT EXISTS test_nkzml8ta ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:51.266834 [ 3800 ] {f5151741-af16-427f-92ee-886b07aba2ed} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:1f3caa26-8d11-435d-95ac-1983d27f3298'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:51.465729 [ 5215 ] {ac1083c7-694b-4ae6-bea7-cabb98139237} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:ef2d5851-4555-4f97-9a52-5bc74bf9aa81'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47168) (comment: 00908_bloom_filter_index.sh) (in query: CREATE DATABASE IF NOT EXISTS test_w7xfjda1 ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:51.478812 [ 5215 ] {ac1083c7-694b-4ae6-bea7-cabb98139237} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:ef2d5851-4555-4f97-9a52-5bc74bf9aa81'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:51.722455 [ 5794 ] {c148c03a-de9e-4690-8384-12bf5243a1a7} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:1b82f539-4421-4907-979a-f9d8ff352674'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:47176) (comment: 01538_fuzz_aggregate.sql) (in query: CREATE DATABASE IF NOT EXISTS test_yfikt868 ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:51.760743 [ 5794 ] {c148c03a-de9e-4690-8384-12bf5243a1a7} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:1b82f539-4421-4907-979a-f9d8ff352674'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:51.955821 [ 5793 ] {730e981a-6de9-43f5-b382-190a30551bd2} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:adff5c0c-e89c-4d20-9c15-d7bae97eca78'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:44424) (comment: 02494_query_cache_secrets.sql) (in query: CREATE DATABASE IF NOT EXISTS test_geja8xed ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:51.962061 [ 5793 ] {730e981a-6de9-43f5-b382-190a30551bd2} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:adff5c0c-e89c-4d20-9c15-d7bae97eca78'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:52.119410 [ 5794 ] {8ea33dc8-9c10-4fe6-a3de-4bff2b84b238} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:0e73b4d3-fa91-45b2-b81d-70015fb01f13'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:44430) (comment: 00573_shard_aggregation_by_empty_set.sql) (in query: CREATE DATABASE IF NOT EXISTS test_mnkeyd2t ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:52.147288 [ 5794 ] {8ea33dc8-9c10-4fe6-a3de-4bff2b84b238} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:0e73b4d3-fa91-45b2-b81d-70015fb01f13'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:52.282287 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:52.354549 [ 5793 ] {755e01d1-eb6b-4e16-ad1c-405063662b3f} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:3f60310f-9057-4dd2-bffd-f23af1db7a53'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:44436) (comment: 02067_lost_part_s3.sql) (in query: CREATE DATABASE IF NOT EXISTS test_wxwldpre ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:52.368426 [ 5793 ] {755e01d1-eb6b-4e16-ad1c-405063662b3f} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:3f60310f-9057-4dd2-bffd-f23af1db7a53'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:52.609828 [ 5215 ] {7280e6de-7ae6-409c-a80d-c38b06f51976} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:836ea3a8-3c8e-47fb-81e3-6d73cd16830c'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:44446) (comment: 01557_field_infinite_convert_to_number.sql) (in query: CREATE DATABASE IF NOT EXISTS test_ltdyxane ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:52.614639 [ 5215 ] {7280e6de-7ae6-409c-a80d-c38b06f51976} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:836ea3a8-3c8e-47fb-81e3-6d73cd16830c'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:52.826748 [ 5794 ] {be0e2049-2a93-4d7f-9812-38be5409d4a7} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:58971fa8-2ff7-44c4-a120-f8c1daa9de28'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:44460) (comment: 02902_topKGeneric_deserialization_memory.sql) (in query: CREATE DATABASE IF NOT EXISTS test_6exnp9kp ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:52.833481 [ 5794 ] {be0e2049-2a93-4d7f-9812-38be5409d4a7} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:58971fa8-2ff7-44c4-a120-f8c1daa9de28'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:52.988520 [ 5215 ] {c87e579c-f14f-414b-82db-da0b6c2d6e17} <Error> executeQuery: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:8e1bc28b-fae6-467c-8a3e-2b08accb479a'. (REPLICA_ALREADY_EXISTS) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:44466) (comment: 02006_client_test_hint_error_name.sql) (in query: CREATE DATABASE IF NOT EXISTS test_goc9j6tf ENGINE=Replicated('/test/db/test_8', 's1', 'r1') ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850

2024.11.05 19:57:52.993758 [ 5215 ] {c87e579c-f14f-414b-82db-da0b6c2d6e17} <Error> DynamicQueryHandler: Code: 253. DB::Exception: Replica r1 of shard s1 of replicated database at /test/db/test_8 already exists. Replica host ID: 'DROPPED', current host ID: '1b19ae151883:9000:8e1bc28b-fae6-467c-8a3e-2b08accb479a'. (REPLICA_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String&, String&, String&, String&, String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type, std::type_identity<String&>::type>, String&, String&, String&, String&, String&) @ 0x000000001a111dda
6. /build/src/Databases/DatabaseReplicated.cpp:375: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x000000001a0edbe9
7. /build/src/Databases/DatabaseReplicated.cpp:543: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0f0a71
8. /build/src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>) @ 0x000000001b0307ba
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:338: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00baef
10. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
11. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
12. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
13. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
14. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
15. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
16. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
17. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
18. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
19. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
20. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
21. ? @ 0x00007f3c91188ac3
22. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:56.231749 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:57.579267 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/und/dhhdqetusoqzxwuvsdszqbdzvaiyt, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 3db700db-5e98-488e-817a-b23ef6d77fa5, upload has finished false.
2024.11.05 19:57:57.579969 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/und/dhhdqetusoqzxwuvsdszqbdzvaiyt, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 3db700db-5e98-488e-817a-b23ef6d77fa5, upload has finished false
2024.11.05 19:57:57.620249 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/und/dhhdqetusoqzxwuvsdszqbdzvaiyt, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 3db700db-5e98-488e-817a-b23ef6d77fa5, upload has finished false
2024.11.05 19:57:57.656741 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 20 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:57:58.892597 [ 5215 ] {c471d354-b6ac-42f9-83f2-02dbbddcbf69} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:52952) (comment: 00180_no_seek_avoiding_when_reading_from_cache.sh) (in query: INSERT INTO hits_s3_sampled SELECT * FROM test.hits_s3 SAMPLE 0.01), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:57:59.222290 [ 5215 ] {c471d354-b6ac-42f9-83f2-02dbbddcbf69} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6

2024.11.05 19:58:03.265932 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/piy/ropupdulcwljykgbotbvtfwbunrbl, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 307f795b-8da3-4931-9359-0ea5fb6ec6f3, upload has finished false.
2024.11.05 19:58:03.266571 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/piy/ropupdulcwljykgbotbvtfwbunrbl, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 307f795b-8da3-4931-9359-0ea5fb6ec6f3, upload has finished false
2024.11.05 19:58:03.298342 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/piy/ropupdulcwljykgbotbvtfwbunrbl, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 307f795b-8da3-4931-9359-0ea5fb6ec6f3, upload has finished false
2024.11.05 19:58:03.357358 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=1): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 12 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:03.533799 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column UTCStartTime): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_6_6_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 0 with max_rows_to_read = 6868): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:05.784295 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:06.369509 [ 3641 ] {} <Error> void DB::SystemLog<DB::PartLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::PartLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:06.678924 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 5 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:06.830022 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:08.677141 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 4 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:09.547300 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 1 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:16.634482 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:22.639068 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {62ec4990-945b-4d8e-9889-ca33c67862f8::202411_1_132_26}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:24.730661 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/fwh/bbcphacxqetqwvjbfwnmkbdsladvm, total size 218103808, count 218103808, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0fd876a5-c179-4d97-a616-b41851187cbb, upload has finished false.
2024.11.05 19:58:24.731972 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/fwh/bbcphacxqetqwvjbfwnmkbdsladvm, total size 218103808, count 218103808, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0fd876a5-c179-4d97-a616-b41851187cbb, upload has finished false
2024.11.05 19:58:24.843259 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/fwh/bbcphacxqetqwvjbfwnmkbdsladvm, total size 218103808, count 218103808, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0fd876a5-c179-4d97-a616-b41851187cbb, upload has finished false
2024.11.05 19:58:24.901912 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 48 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:27.391099 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {29fdad69-1244-4c25-af3c-70c43e910f15::202411_1_201_25}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:28.829776 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 6 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:29.342013 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {29fdad69-1244-4c25-af3c-70c43e910f15::202411_1_201_25}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column thread_name): (while reading from part /var/lib/clickhouse/disks/s3/store/29f/29fdad69-1244-4c25-af3c-70c43e910f15/202411_1_140_24/ in table system.text_log (29fdad69-1244-4c25-af3c-70c43e910f15) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 8192): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/SerializationLowCardinality.cpp:62: DB::SerializationLowCardinality::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x000000001a0107d2
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:29.449789 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:31.852852 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:32.408283 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {62ec4990-945b-4d8e-9889-ca33c67862f8::202411_1_133_26}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
25. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
26. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
27. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
28. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:32.942201 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0e641ae6-a90e-4cec-8ebd-a21f62e7934a::202411_1_127_28}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
24. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
25. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
26. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
27. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:34.223481 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {29fdad69-1244-4c25-af3c-70c43e910f15::202411_202_207_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:34.409691 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/syn/yrlxtyujwdthaoxzyhsgwngcrffms, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 2cdcc144-9a92-4ab3-8bb9-dca34772010e, upload has finished false.
2024.11.05 19:58:34.412162 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/syn/yrlxtyujwdthaoxzyhsgwngcrffms, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 2cdcc144-9a92-4ab3-8bb9-dca34772010e, upload has finished false
2024.11.05 19:58:34.435167 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/syn/yrlxtyujwdthaoxzyhsgwngcrffms, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 2cdcc144-9a92-4ab3-8bb9-dca34772010e, upload has finished false
2024.11.05 19:58:34.491881 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 13 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:35.526968 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:35.752193 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:36.800675 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:37.784619 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 4 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:40.239494 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {c04f5a1b-d832-4de4-96b0-dec4e591ea83::202411_1_117_24}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:42.251876 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 9 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:42.871227 [ 3642 ] {} <Error> void DB::SystemLog<DB::MetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::MetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:44.517330 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column PageViews): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_2_2_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 0 with max_rows_to_read = 6592): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:45.585450 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 8 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:47.556420 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:48.065150 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:48.515037 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:58:55.167922 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:00.036404 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/ohi/wekwhekaojgrmzvbrwgxittvmylba, total size 218103808, count 218103808, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 22bf15e9-7e12-4545-8859-43f0c679dc49, upload has finished false.
2024.11.05 19:59:00.038980 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/ohi/wekwhekaojgrmzvbrwgxittvmylba, total size 218103808, count 218103808, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 22bf15e9-7e12-4545-8859-43f0c679dc49, upload has finished false
2024.11.05 19:59:00.115018 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/ohi/wekwhekaojgrmzvbrwgxittvmylba, total size 218103808, count 218103808, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 22bf15e9-7e12-4545-8859-43f0c679dc49, upload has finished false
2024.11.05 19:59:00.165847 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 46 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:00.274918 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:03.802850 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 6 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:06.132040 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:08.680326 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:13.859258 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/gks/cymdzpvtbiecpatnoptyspexjbdve, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id bba1bbc3-ef6f-454e-83a1-edf80b45669d, upload has finished false.
2024.11.05 19:59:13.860007 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/gks/cymdzpvtbiecpatnoptyspexjbdve, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id bba1bbc3-ef6f-454e-83a1-edf80b45669d, upload has finished false
2024.11.05 19:59:13.931696 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/gks/cymdzpvtbiecpatnoptyspexjbdve, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id bba1bbc3-ef6f-454e-83a1-edf80b45669d, upload has finished false
2024.11.05 19:59:14.028638 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 33 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:19.585499 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/uzc/thawxnckrvrbfcyjrkprvobyinfih, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8ebf8d38-b8b4-4a32-95d8-ff2949207f56, upload has finished false.
2024.11.05 19:59:19.586811 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/uzc/thawxnckrvrbfcyjrkprvobyinfih, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8ebf8d38-b8b4-4a32-95d8-ff2949207f56, upload has finished false
2024.11.05 19:59:19.615096 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/uzc/thawxnckrvrbfcyjrkprvobyinfih, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8ebf8d38-b8b4-4a32-95d8-ff2949207f56, upload has finished false
2024.11.05 19:59:19.634079 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 14 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:25.714031 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/nlt/irpebqhyqstfgaqusqrmkohsslesz, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 72df04b0-1ced-46bd-acbd-72cd73d850a7, upload has finished false.
2024.11.05 19:59:25.717808 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/nlt/irpebqhyqstfgaqusqrmkohsslesz, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 72df04b0-1ced-46bd-acbd-72cd73d850a7, upload has finished false
2024.11.05 19:59:25.762041 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/nlt/irpebqhyqstfgaqusqrmkohsslesz, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 72df04b0-1ced-46bd-acbd-72cd73d850a7, upload has finished false
2024.11.05 19:59:25.783588 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 18 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:29.060179 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {dbd666d3-2d23-40c6-8136-6ca2be336b4d::202411_1_109_20}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:29.591967 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 8 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:30.082746 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:32.809959 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:34.321422 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 11 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:38.737272 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:39.091336 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/dnm/edvryukcqyknbdybblofgvwvxcoto, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id bfb356d6-e6d7-4b55-8877-44355d537683, upload has finished false.
2024.11.05 19:59:39.091835 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/dnm/edvryukcqyknbdybblofgvwvxcoto, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id bfb356d6-e6d7-4b55-8877-44355d537683, upload has finished false
2024.11.05 19:59:39.111013 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/dnm/edvryukcqyknbdybblofgvwvxcoto, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id bfb356d6-e6d7-4b55-8877-44355d537683, upload has finished false
2024.11.05 19:59:39.165828 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:567: DB::WriteBufferFromS3::writePart(DB::WriteBufferFromS3::PartData&&) @ 0x0000000019914d6e
18. /build/src/IO/WriteBufferFromS3.cpp:370: DB::WriteBufferFromS3::writeMultipartUpload() @ 0x0000000019908eb6
19. /build/src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::nextImpl() @ 0x000000001990857e
20. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
21. /build/src/IO/WriteBufferFromFileDecorator.cpp:71: DB::WriteBufferFromFileDecorator::nextImpl() @ 0x000000001a3593ad
22. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
23. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
24. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
25. /build/src/IO/WriteBuffer.h:76: DB::WriteBuffer::nextIfAtEnd() @ 0x000000000a931978
26. /build/src/IO/WriteBuffer.h:91: DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000a9317e0
27. /build/src/Compression/CompressedWriteBuffer.cpp:58: DB::CompressedWriteBuffer::nextImpl() @ 0x0000000019ddd7ed
28. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
29. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
30. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
31. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:350: DB::MergeTreeDataPartWriterWide::getCurrentMarksForColumn(DB::NameAndTypePair const&, std::set<String, std::less<String>, std::allocator<String>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ccf5839
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:40.012572 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:41.816191 [ 5794 ] {02c2f338-7924-45c6-91e5-9a2abb4597c5} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1011, jobs=896). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:40234) (comment: 01058_window_view_event_hop_to_strict_asc.sh) (in query: SELECT count(*) FROM dst), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001d5394f3
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:112: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000001d539234
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000001d5395e9
12. /build/src/Server/TCPHandler.cpp:1023: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b57f
13. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:59:41.847631 [ 5794 ] {02c2f338-7924-45c6-91e5-9a2abb4597c5} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1011, jobs=896). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001d5394f3
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:112: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000001d539234
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000001d5395e9
12. /build/src/Server/TCPHandler.cpp:1023: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b57f
13. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 19:59:43.050663 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0aef53a4-984b-414b-bb72-6836bf38bb48::202411_1_137_28}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:274: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3abc4
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:44.618089 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=1): (while reading column Referer): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_3_3_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 26 with max_rows_to_read = 6608): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:48.228453 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:59:52.522271 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0aef53a4-984b-414b-bb72-6836bf38bb48::202411_1_143_29}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:03.844942 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:04.642811 [ 5794 ] {98cb0ce4-4be4-4d30-9759-79dd1a61c47e} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1011, jobs=896). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:39012) (comment: 01058_window_view_event_hop_to_strict_asc.sh) (in query: SELECT count(*) FROM dst), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001d5394f3
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:112: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000001d539234
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000001d5395e9
12. /build/src/Server/TCPHandler.cpp:1023: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b57f
13. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 20:00:04.659203 [ 5794 ] {98cb0ce4-4be4-4d30-9759-79dd1a61c47e} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=1011, jobs=896). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133dccad
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d4c58
8. /build/src/Common/ThreadPool.cpp:338: ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, Priority, unsigned long, bool) @ 0x00000000133d5510
9. /build/src/Common/ThreadPool.h:205: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001d5394f3
10. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:112: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x000000001d539234
11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:138: DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x000000001d5395e9
12. /build/src/Server/TCPHandler.cpp:1023: DB::TCPHandler::processOrdinaryQuery() @ 0x000000001d47b57f
13. /build/src/Server/TCPHandler.cpp:539: DB::TCPHandler::runImpl() @ 0x000000001d47085b
14. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 20:00:09.145548 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/ogm/rpnluxgwdshvenghwjerkwedwkhqs, total size 501975500, count 501975500, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id f2e4e02d-bd4f-49ae-b465-fd24423a757a, upload has finished false.
2024.11.05 20:00:09.149335 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/ogm/rpnluxgwdshvenghwjerkwedwkhqs, total size 501975500, count 501975500, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id f2e4e02d-bd4f-49ae-b465-fd24423a757a, upload has finished false
2024.11.05 20:00:09.301476 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/ogm/rpnluxgwdshvenghwjerkwedwkhqs, total size 501975500, count 501975500, hidden_size 0, offset 0, with pool: true, prefinalized true, finalized false, upload id f2e4e02d-bd4f-49ae-b465-fd24423a757a, upload has finished false
2024.11.05 20:00:09.316685 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:45: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce610d
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
24. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
25. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
26. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
27. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
28. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:11.168439 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:12.248318 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 1 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:14.246456 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {ffab63a2-a3d3-43ea-ab17-7d5025e1f007::202411_1_40_8}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:17.306722 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 11 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:18.079668 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:19.192927 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 2 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:21.334331 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 3 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:21.754922 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {c04f5a1b-d832-4de4-96b0-dec4e591ea83::202411_1_126_28}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
25. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
26. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
27. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
28. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:22.007592 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:30.781854 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/amr/wgcdgaylcakpcpckmacrcxmgsujxh, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 04e497ba-e0eb-49ee-b5b3-edb968ec6f68, upload has finished false.
2024.11.05 20:00:30.783962 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/amr/wgcdgaylcakpcpckmacrcxmgsujxh, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 04e497ba-e0eb-49ee-b5b3-edb968ec6f68, upload has finished false
2024.11.05 20:00:30.833522 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/amr/wgcdgaylcakpcpckmacrcxmgsujxh, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 04e497ba-e0eb-49ee-b5b3-edb968ec6f68, upload has finished false
2024.11.05 20:00:30.857264 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:567: DB::WriteBufferFromS3::writePart(DB::WriteBufferFromS3::PartData&&) @ 0x0000000019914d6e
18. /build/src/IO/WriteBufferFromS3.cpp:370: DB::WriteBufferFromS3::writeMultipartUpload() @ 0x0000000019908eb6
19. /build/src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::nextImpl() @ 0x000000001990857e
20. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
21. /build/src/IO/WriteBufferFromFileDecorator.cpp:71: DB::WriteBufferFromFileDecorator::nextImpl() @ 0x000000001a3593ad
22. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
23. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
24. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
25. /build/src/IO/WriteBuffer.h:76: DB::WriteBuffer::nextIfAtEnd() @ 0x000000000a931978
26. /build/src/IO/WriteBuffer.h:91: DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000a9317e0
27. /build/src/Compression/CompressedWriteBuffer.cpp:58: DB::CompressedWriteBuffer::nextImpl() @ 0x0000000019ddd7ed
28. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
29. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
30. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
31. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:350: DB::MergeTreeDataPartWriterWide::getCurrentMarksForColumn(DB::NameAndTypePair const&, std::set<String, std::less<String>, std::allocator<String>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ccf5839
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:33.536395 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 6 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:34.137053 [ 3580 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:35.579845 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:36.936559 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:37.218782 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 6 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:38.137385 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column n): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:40.441080 [ 3612 ] {} <Warning> test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21): Removing temporary directory /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/tmp_merge_201403_1_6_1/
2024.11.05 20:00:40.441679 [ 3612 ] {} <Warning> test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/tmp_merge_201403_1_6_1/
2024.11.05 20:00:40.600323 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:40.614050 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column Referer): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_3_3_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 17 with max_rows_to_read = 6608): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:43.111531 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=1): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 12 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:44.630216 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:45.055184 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:46.043914 [ 3609 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Removing temporary directory /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 20:00:46.044630 [ 3609 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 20:00:46.255065 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 8 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:54.473505 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:54.725215 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:00:56.172433 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/hux/zmugxnsrqanxblrkvtibuorkphnwy, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id fca5592c-778a-457d-97ff-1846aa96400a, upload has finished false.
2024.11.05 20:00:56.173394 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/hux/zmugxnsrqanxblrkvtibuorkphnwy, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id fca5592c-778a-457d-97ff-1846aa96400a, upload has finished false
2024.11.05 20:00:56.228777 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/hux/zmugxnsrqanxblrkvtibuorkphnwy, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id fca5592c-778a-457d-97ff-1846aa96400a, upload has finished false
2024.11.05 20:00:56.314933 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 28 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:02.590300 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/krp/sbzdngtkaedkdbzkuhhjxalmccbma, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id d02f1ffd-9d74-4f19-9409-5857ae893994, upload has finished false.
2024.11.05 20:01:02.598600 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/krp/sbzdngtkaedkdbzkuhhjxalmccbma, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id d02f1ffd-9d74-4f19-9409-5857ae893994, upload has finished false
2024.11.05 20:01:02.641849 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/krp/sbzdngtkaedkdbzkuhhjxalmccbma, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id d02f1ffd-9d74-4f19-9409-5857ae893994, upload has finished false
2024.11.05 20:01:02.731286 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 16 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:04.656933 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:04.915150 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0a233324-63e4-4351-8154-d23ef4927e18::202411_1_136_27}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:05.015121 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:05.855844 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 1 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:07.273772 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:09.365905 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {c04f5a1b-d832-4de4-96b0-dec4e591ea83::202411_1_130_32}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:12.292070 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/nff/qgntyqoudwsudzdgvuqfmztnbfxit, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 85df0f54-88ac-43e9-97ff-e5596f0e9a88, upload has finished false.
2024.11.05 20:01:12.293472 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/nff/qgntyqoudwsudzdgvuqfmztnbfxit, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 85df0f54-88ac-43e9-97ff-e5596f0e9a88, upload has finished false
2024.11.05 20:01:12.297228 [ 3615 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Removing temporary directory /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 20:01:12.300478 [ 3615 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 20:01:12.341572 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/nff/qgntyqoudwsudzdgvuqfmztnbfxit, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 85df0f54-88ac-43e9-97ff-e5596f0e9a88, upload has finished false
2024.11.05 20:01:12.381773 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:567: DB::WriteBufferFromS3::writePart(DB::WriteBufferFromS3::PartData&&) @ 0x0000000019914d6e
18. /build/src/IO/WriteBufferFromS3.cpp:370: DB::WriteBufferFromS3::writeMultipartUpload() @ 0x0000000019908eb6
19. /build/src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::nextImpl() @ 0x000000001990857e
20. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
21. /build/src/IO/WriteBufferFromFileDecorator.cpp:71: DB::WriteBufferFromFileDecorator::nextImpl() @ 0x000000001a3593ad
22. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
23. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
24. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
25. /build/src/IO/WriteBuffer.h:76: DB::WriteBuffer::nextIfAtEnd() @ 0x000000000a931978
26. /build/src/IO/WriteBuffer.h:91: DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000a9317e0
27. /build/src/Compression/CompressedWriteBuffer.cpp:58: DB::CompressedWriteBuffer::nextImpl() @ 0x0000000019ddd7ed
28. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
29. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
30. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
31. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:350: DB::MergeTreeDataPartWriterWide::getCurrentMarksForColumn(DB::NameAndTypePair const&, std::set<String, std::less<String>, std::allocator<String>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ccf5839
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:13.511501 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:18.479031 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/hwr/nvyzwzzgcrrcalwjsmggzkxpkgkrt, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 91b26955-2594-45c3-bee8-a4e243265551, upload has finished false.
2024.11.05 20:01:18.479790 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/hwr/nvyzwzzgcrrcalwjsmggzkxpkgkrt, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 91b26955-2594-45c3-bee8-a4e243265551, upload has finished false
2024.11.05 20:01:18.544224 [ 3579 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/hwr/nvyzwzzgcrrcalwjsmggzkxpkgkrt, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 91b26955-2594-45c3-bee8-a4e243265551, upload has finished false
2024.11.05 20:01:18.560502 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:567: DB::WriteBufferFromS3::writePart(DB::WriteBufferFromS3::PartData&&) @ 0x0000000019914d6e
18. /build/src/IO/WriteBufferFromS3.cpp:370: DB::WriteBufferFromS3::writeMultipartUpload() @ 0x0000000019908eb6
19. /build/src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::nextImpl() @ 0x000000001990857e
20. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
21. /build/src/IO/WriteBufferFromFileDecorator.cpp:71: DB::WriteBufferFromFileDecorator::nextImpl() @ 0x000000001a3593ad
22. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
23. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
24. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
25. /build/src/IO/WriteBuffer.h:76: DB::WriteBuffer::nextIfAtEnd() @ 0x000000000a931978
26. /build/src/IO/WriteBuffer.h:91: DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000a9317e0
27. /build/src/Compression/CompressedWriteBuffer.cpp:58: DB::CompressedWriteBuffer::nextImpl() @ 0x0000000019ddd7ed
28. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
29. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
30. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
31. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:350: DB::MergeTreeDataPartWriterWide::getCurrentMarksForColumn(DB::NameAndTypePair const&, std::set<String, std::less<String>, std::allocator<String>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ccf5839
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:23.196912 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 9 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:28.987057 [ 3611 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Removing temporary directory /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 20:01:28.987875 [ 3611 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 20:01:29.224641 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/uyq/ioetlfwiyhuufqqzthiiqfpcueftx, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 4490668d-ce7a-4f95-875f-8297961c48d0, upload has finished false.
2024.11.05 20:01:29.233879 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/uyq/ioetlfwiyhuufqqzthiiqfpcueftx, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 4490668d-ce7a-4f95-875f-8297961c48d0, upload has finished false
2024.11.05 20:01:29.269631 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/uyq/ioetlfwiyhuufqqzthiiqfpcueftx, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 4490668d-ce7a-4f95-875f-8297961c48d0, upload has finished false
2024.11.05 20:01:29.330245 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 16 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:29.648473 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:29.825753 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:37.208827 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/uul/ogwoydbcqroxeiwtdpqdhnfuujlyy, total size 100663296, count 100663296, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ea5b6a24-fd3d-4993-aa85-b66d015181c7, upload has finished false.
2024.11.05 20:01:37.210914 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/uul/ogwoydbcqroxeiwtdpqdhnfuujlyy, total size 100663296, count 100663296, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ea5b6a24-fd3d-4993-aa85-b66d015181c7, upload has finished false
2024.11.05 20:01:37.252741 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/uul/ogwoydbcqroxeiwtdpqdhnfuujlyy, total size 100663296, count 100663296, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ea5b6a24-fd3d-4993-aa85-b66d015181c7, upload has finished false
2024.11.05 20:01:37.307113 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 22 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:37.715265 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0aef53a4-984b-414b-bb72-6836bf38bb48::202411_1_150_32}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:41.012886 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4ff3d7e3-1bf7-4d39-8a23-c88787b766d0::202411_1_147_22}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:43: DB::MergeTreeDataPartWriterOnDisk::Stream<false>::preFinalize() @ 0x000000001cce60f4
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:644: DB::MergeTreeDataPartWriterWide::fillDataChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3583
22. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:0: DB::MergeTreeDataPartWriterWide::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccf3a7e
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
24. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
25. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
26. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
27. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
28. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:43.243772 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0a233324-63e4-4351-8154-d23ef4927e18::202411_1_141_28}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:43.890004 [ 3581 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key s3/vgf/cdncifdrcusgogjieirciysxuzago, total size 100663296, count 100663296, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id db5143ba-29a4-425e-99f9-35b3343df663, upload has finished false.
2024.11.05 20:01:43.897256 [ 3581 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key s3/vgf/cdncifdrcusgogjieirciysxuzago, total size 100663296, count 100663296, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id db5143ba-29a4-425e-99f9-35b3343df663, upload has finished false
2024.11.05 20:01:43.938955 [ 3581 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key s3/vgf/cdncifdrcusgogjieirciysxuzago, total size 100663296, count 100663296, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id db5143ba-29a4-425e-99f9-35b3343df663, upload has finished false
2024.11.05 20:01:44.217309 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=2): (while reading column Referer): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_6_6_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 7 with max_rows_to_read = 6868): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:45.408426 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/qir/falbabnkfjhthoturwdzorttypknr, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6d029a73-9dc6-4b1f-889e-936548f5e868, upload has finished false.
2024.11.05 20:01:45.409325 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/qir/falbabnkfjhthoturwdzorttypknr, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6d029a73-9dc6-4b1f-889e-936548f5e868, upload has finished false
2024.11.05 20:01:45.440632 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/qir/falbabnkfjhthoturwdzorttypknr, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6d029a73-9dc6-4b1f-889e-936548f5e868, upload has finished false
2024.11.05 20:01:45.684402 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 19 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:47.015790 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:51.341997 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/fsz/vqetzlqkkuoxcisgytxiiyoxnxncf, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 46ab7c84-f04d-4356-a238-c0475462bc1c, upload has finished false.
2024.11.05 20:01:51.345686 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/fsz/vqetzlqkkuoxcisgytxiiyoxnxncf, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 46ab7c84-f04d-4356-a238-c0475462bc1c, upload has finished false
2024.11.05 20:01:51.395250 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/fsz/vqetzlqkkuoxcisgytxiiyoxnxncf, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 46ab7c84-f04d-4356-a238-c0475462bc1c, upload has finished false
2024.11.05 20:01:51.403618 [ 3614 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Removing temporary directory /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 20:01:51.404450 [ 3614 ] {} <Warning> test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885): Since zero-copy replication is enabled we are not going to remove blobs from shared storage for /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/tmp_merge_all_1_5_1/
2024.11.05 20:01:51.434546 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:567: DB::WriteBufferFromS3::writePart(DB::WriteBufferFromS3::PartData&&) @ 0x0000000019914d6e
18. /build/src/IO/WriteBufferFromS3.cpp:370: DB::WriteBufferFromS3::writeMultipartUpload() @ 0x0000000019908eb6
19. /build/src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::nextImpl() @ 0x000000001990857e
20. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
21. /build/src/IO/WriteBufferFromFileDecorator.cpp:71: DB::WriteBufferFromFileDecorator::nextImpl() @ 0x000000001a3593ad
22. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
23. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
24. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
25. /build/src/IO/WriteBuffer.h:76: DB::WriteBuffer::nextIfAtEnd() @ 0x000000000a931978
26. /build/src/IO/WriteBuffer.h:91: DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000a9317e0
27. /build/src/Compression/CompressedWriteBuffer.cpp:58: DB::CompressedWriteBuffer::nextImpl() @ 0x0000000019ddd7ed
28. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
29. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
30. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
31. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:350: DB::MergeTreeDataPartWriterWide::getCurrentMarksForColumn(DB::NameAndTypePair const&, std::set<String, std::less<String>, std::allocator<String>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ccf5839
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:01:57.255626 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {c04f5a1b-d832-4de4-96b0-dec4e591ea83::202411_1_134_36}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
25. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
26. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
27. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
28. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
31. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:00.281484 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:01.680130 [ 3583 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/mkm/rpttjyxvhelkqsuvmhsdolgmrbvrq, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f4fa93d1-7165-4aa6-9391-6c0bf51b614f, upload has finished false.
2024.11.05 20:02:01.682052 [ 3583 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/mkm/rpttjyxvhelkqsuvmhsdolgmrbvrq, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f4fa93d1-7165-4aa6-9391-6c0bf51b614f, upload has finished false
2024.11.05 20:02:01.736886 [ 3583 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/mkm/rpttjyxvhelkqsuvmhsdolgmrbvrq, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f4fa93d1-7165-4aa6-9391-6c0bf51b614f, upload has finished false
2024.11.05 20:02:01.765714 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 31 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:02.331754 [ 3642 ] {} <Error> void DB::SystemLog<DB::MetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::MetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:02.565129 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:04.152838 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 3 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:11.233750 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:13.280163 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/pvo/xzjehcfuakaolxaahymermxfyuasd, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b1e7947b-14d9-4d22-a7f9-eb778c8c9339, upload has finished false.
2024.11.05 20:02:13.281204 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/pvo/xzjehcfuakaolxaahymermxfyuasd, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b1e7947b-14d9-4d22-a7f9-eb778c8c9339, upload has finished false
2024.11.05 20:02:13.344350 [ 3572 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/pvo/xzjehcfuakaolxaahymermxfyuasd, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id b1e7947b-14d9-4d22-a7f9-eb778c8c9339, upload has finished false
2024.11.05 20:02:13.375761 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 30 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:15.808651 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:19.227693 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:21.389012 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {c04f5a1b-d832-4de4-96b0-dec4e591ea83::202411_1_136_38}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
24. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
25. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
26. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
27. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:27.343293 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/gqp/btlayjsgvdbtscukggqhdcdmbptcf, total size 201326592, count 201326592, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id deef796b-df2b-481c-8773-9a4b8255b2f1, upload has finished false.
2024.11.05 20:02:27.345587 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/gqp/btlayjsgvdbtscukggqhdcdmbptcf, total size 201326592, count 201326592, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id deef796b-df2b-481c-8773-9a4b8255b2f1, upload has finished false
2024.11.05 20:02:27.424255 [ 3577 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/gqp/btlayjsgvdbtscukggqhdcdmbptcf, total size 201326592, count 201326592, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id deef796b-df2b-481c-8773-9a4b8255b2f1, upload has finished false
2024.11.05 20:02:27.488687 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 45 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:29.430274 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 1 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:30.286496 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 1 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:35.137566 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:40.224661 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/gxz/phoybkxhtlhrqkosjrgjkoxveicez, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id a1c1bbc0-f918-49a1-b425-e5a9026289b1, upload has finished false.
2024.11.05 20:02:40.229214 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/gxz/phoybkxhtlhrqkosjrgjkoxveicez, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id a1c1bbc0-f918-49a1-b425-e5a9026289b1, upload has finished false
2024.11.05 20:02:40.291417 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/gxz/phoybkxhtlhrqkosjrgjkoxveicez, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id a1c1bbc0-f918-49a1-b425-e5a9026289b1, upload has finished false
2024.11.05 20:02:40.481985 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 34 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:41.539551 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=6). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:42.360007 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:42.517849 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:43.961202 [ 3571 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key s3/nfy/mcmmhroeaalexkorsuzfanfczbxgw, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ddd85648-e4dc-4347-8ac3-6288fdd75bf4, upload has finished false.
2024.11.05 20:02:43.967400 [ 3571 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key s3/nfy/mcmmhroeaalexkorsuzfanfczbxgw, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ddd85648-e4dc-4347-8ac3-6288fdd75bf4, upload has finished false
2024.11.05 20:02:44.014021 [ 3571 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key s3/nfy/mcmmhroeaalexkorsuzfanfczbxgw, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id ddd85648-e4dc-4347-8ac3-6288fdd75bf4, upload has finished false
2024.11.05 20:02:44.288236 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column Referer): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_3_3_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 38 with max_rows_to_read = 6608): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:44.408359 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:48.815371 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=1): (while reading column StartDate): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_4_4_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 0 with max_rows_to_read = 6741): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:52.368456 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {29fdad69-1244-4c25-af3c-70c43e910f15::202411_202_258_11}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:53.719769 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {c04f5a1b-d832-4de4-96b0-dec4e591ea83::202411_1_138_40}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:53.931118 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0e641ae6-a90e-4cec-8ebd-a21f62e7934a::202411_1_157_58}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:54.319098 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/ozm/btcgenctfxhcfotvyxuthtoebgpgc, total size 184549376, count 184549376, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f33e1d6a-12a8-4ab9-b3f3-5e527a04edc6, upload has finished false.
2024.11.05 20:02:54.326117 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/ozm/btcgenctfxhcfotvyxuthtoebgpgc, total size 184549376, count 184549376, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f33e1d6a-12a8-4ab9-b3f3-5e527a04edc6, upload has finished false
2024.11.05 20:02:54.402908 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/ozm/btcgenctfxhcfotvyxuthtoebgpgc, total size 184549376, count 184549376, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f33e1d6a-12a8-4ab9-b3f3-5e527a04edc6, upload has finished false
2024.11.05 20:02:54.500175 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 41 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:02:59.209773 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:01.361370 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:04.577627 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/til/ttpgzyafjsumxtbnknkbzduwegswt, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c7b0b45e-ea39-4251-ba75-b08592dc8b30, upload has finished false.
2024.11.05 20:03:04.582455 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/til/ttpgzyafjsumxtbnknkbzduwegswt, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c7b0b45e-ea39-4251-ba75-b08592dc8b30, upload has finished false
2024.11.05 20:03:04.639660 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/til/ttpgzyafjsumxtbnknkbzduwegswt, total size 134217728, count 134217728, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id c7b0b45e-ea39-4251-ba75-b08592dc8b30, upload has finished false
2024.11.05 20:03:04.661240 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:567: DB::WriteBufferFromS3::writePart(DB::WriteBufferFromS3::PartData&&) @ 0x0000000019914d6e
18. /build/src/IO/WriteBufferFromS3.cpp:370: DB::WriteBufferFromS3::writeMultipartUpload() @ 0x0000000019908eb6
19. /build/src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::nextImpl() @ 0x000000001990857e
20. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
21. /build/src/IO/WriteBufferFromFileDecorator.cpp:71: DB::WriteBufferFromFileDecorator::nextImpl() @ 0x000000001a3593ad
22. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
23. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
24. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
25. /build/src/IO/WriteBuffer.h:76: DB::WriteBuffer::nextIfAtEnd() @ 0x000000000a931978
26. /build/src/IO/WriteBuffer.h:91: DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000a9317e0
27. /build/src/Compression/CompressedWriteBuffer.cpp:58: DB::CompressedWriteBuffer::nextImpl() @ 0x0000000019ddd7ed
28. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
29. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
30. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
31. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:350: DB::MergeTreeDataPartWriterWide::getCurrentMarksForColumn(DB::NameAndTypePair const&, std::set<String, std::less<String>, std::allocator<String>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ccf5839
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:07.092448 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 6 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:08.002056 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 1 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:12.069804 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 10 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:21.363428 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:22.741850 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/ixd/poxvllhbtpbruoykvpdssyhzflfzb, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id a5244ae8-3fb2-480e-b959-e12f50414653, upload has finished false.
2024.11.05 20:03:22.742557 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/ixd/poxvllhbtpbruoykvpdssyhzflfzb, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id a5244ae8-3fb2-480e-b959-e12f50414653, upload has finished false
2024.11.05 20:03:22.804749 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/ixd/poxvllhbtpbruoykvpdssyhzflfzb, total size 150994944, count 150994944, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id a5244ae8-3fb2-480e-b959-e12f50414653, upload has finished false
2024.11.05 20:03:22.853960 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 34 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:31.034331 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {c04f5a1b-d832-4de4-96b0-dec4e591ea83::202411_1_142_44}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:34.809767 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:35.171230 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:39.029951 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:40.918893 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/mxu/elkawogljgbnbbawrbxsrislkzcgk, total size 251658240, count 251658240, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f8ea1120-7883-4279-b329-d2a7cd627da4, upload has finished false.
2024.11.05 20:03:40.919698 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/mxu/elkawogljgbnbbawrbxsrislkzcgk, total size 251658240, count 251658240, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f8ea1120-7883-4279-b329-d2a7cd627da4, upload has finished false
2024.11.05 20:03:41.112587 [ 3576 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/mxu/elkawogljgbnbbawrbxsrislkzcgk, total size 251658240, count 251658240, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f8ea1120-7883-4279-b329-d2a7cd627da4, upload has finished false
2024.11.05 20:03:41.140638 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 54 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:41.300284 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column Referer): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_1_1_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 22 with max_rows_to_read = 6658): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:42.689113 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 1 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:43.179774 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:44.856745 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=2): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 3 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:03:50.841484 [ 3641 ] {} <Error> void DB::SystemLog<DB::PartLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::PartLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:05.226948 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/vnc/hjsgvmpbwbmeakijqfczwavwzgvne, total size 318767104, count 318767104, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id fd9f09f1-fd77-48fb-ab4f-a53767391bc9, upload has finished false.
2024.11.05 20:04:05.227658 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/vnc/hjsgvmpbwbmeakijqfczwavwzgvne, total size 318767104, count 318767104, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id fd9f09f1-fd77-48fb-ab4f-a53767391bc9, upload has finished false
2024.11.05 20:04:05.336342 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/vnc/hjsgvmpbwbmeakijqfczwavwzgvne, total size 318767104, count 318767104, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id fd9f09f1-fd77-48fb-ab4f-a53767391bc9, upload has finished false
2024.11.05 20:04:05.364492 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 69 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:08.637198 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column IsNew): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_3_3_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 0 with max_rows_to_read = 6608): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:09.606982 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 9 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:11.335720 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 1 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:13.718794 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 2 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:25.973904 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/oom/wpecxrdhusrzpawfiixzcixqvkbqn, total size 167772160, count 167772160, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f62a3fd4-4c1b-45ff-b559-cbca46d5cad3, upload has finished false.
2024.11.05 20:04:25.977596 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/oom/wpecxrdhusrzpawfiixzcixqvkbqn, total size 167772160, count 167772160, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f62a3fd4-4c1b-45ff-b559-cbca46d5cad3, upload has finished false
2024.11.05 20:04:26.054197 [ 3578 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/oom/wpecxrdhusrzpawfiixzcixqvkbqn, total size 167772160, count 167772160, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id f62a3fd4-4c1b-45ff-b559-cbca46d5cad3, upload has finished false
2024.11.05 20:04:26.149272 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 37 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:26.443855 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:310: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b386
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:33.663180 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/bdf/czzetudycyfmlatumvakzkxfgpkvj, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 006e3b1f-cead-470e-9013-05a949aacc8d, upload has finished false.
2024.11.05 20:04:33.664291 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/bdf/czzetudycyfmlatumvakzkxfgpkvj, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 006e3b1f-cead-470e-9013-05a949aacc8d, upload has finished false
2024.11.05 20:04:33.754105 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/bdf/czzetudycyfmlatumvakzkxfgpkvj, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 006e3b1f-cead-470e-9013-05a949aacc8d, upload has finished false
2024.11.05 20:04:33.792743 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:567: DB::WriteBufferFromS3::writePart(DB::WriteBufferFromS3::PartData&&) @ 0x0000000019914d6e
18. /build/src/IO/WriteBufferFromS3.cpp:370: DB::WriteBufferFromS3::writeMultipartUpload() @ 0x0000000019908eb6
19. /build/src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::nextImpl() @ 0x000000001990857e
20. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
21. /build/src/IO/WriteBufferFromFileDecorator.cpp:71: DB::WriteBufferFromFileDecorator::nextImpl() @ 0x000000001a3593ad
22. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
23. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
24. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
25. /build/src/IO/WriteBuffer.h:76: DB::WriteBuffer::nextIfAtEnd() @ 0x000000000a931978
26. /build/src/IO/WriteBuffer.h:91: DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000a9317e0
27. /build/src/Compression/CompressedWriteBuffer.cpp:58: DB::CompressedWriteBuffer::nextImpl() @ 0x0000000019ddd7ed
28. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
29. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
30. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
31. /build/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:350: DB::MergeTreeDataPartWriterWide::getCurrentMarksForColumn(DB::NameAndTypePair const&, std::set<String, std::less<String>, std::allocator<String>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ccf5839
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:34.582061 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:35.522538 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {ffab63a2-a3d3-43ea-ab17-7d5025e1f007::202411_1_50_10}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:37.228273 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0aef53a4-984b-414b-bb72-6836bf38bb48::202411_1_171_43}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:37.243581 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:41.045194 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:42.216833 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreePartition.cpp:441: DB::MergeTreePartition::store(DB::Block const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&, DB::WriteSettings const&) const @ 0x000000001ce306c6
21. /build/src/Storages/MergeTree/MergeTreePartition.cpp:421: DB::MergeTreePartition::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001ce2f878
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:239: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a3b5
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:43.186029 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0aef53a4-984b-414b-bb72-6836bf38bb48::202411_1_177_44}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:46.387052 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {dbd666d3-2d23-40c6-8136-6ca2be336b4d::202411_1_139_26}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:47.953887 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {dbd666d3-2d23-40c6-8136-6ca2be336b4d::202411_1_139_26}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:49.009366 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:49.828421 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:51.039074 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:51.691318 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:51.737915 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/pfq/wgoewcopowklsblytmcggodvssrzf, total size 301989888, count 301989888, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id d79f1cf3-d899-4466-b564-cc85e20471e8, upload has finished false.
2024.11.05 20:04:51.739124 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/pfq/wgoewcopowklsblytmcggodvssrzf, total size 301989888, count 301989888, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id d79f1cf3-d899-4466-b564-cc85e20471e8, upload has finished false
2024.11.05 20:04:51.826679 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/pfq/wgoewcopowklsblytmcggodvssrzf, total size 301989888, count 301989888, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id d79f1cf3-d899-4466-b564-cc85e20471e8, upload has finished false
2024.11.05 20:04:51.967065 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 63 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:55.509127 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 8 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:04:56.768126 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:00.571934 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/ytp/ttlkrwicyopowirnegphrffklwedb, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6473be5e-cd7f-4871-913e-1cd4876a5d55, upload has finished false.
2024.11.05 20:05:00.577393 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/ytp/ttlkrwicyopowirnegphrffklwedb, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6473be5e-cd7f-4871-913e-1cd4876a5d55, upload has finished false
2024.11.05 20:05:00.617980 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/ytp/ttlkrwicyopowirnegphrffklwedb, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6473be5e-cd7f-4871-913e-1cd4876a5d55, upload has finished false
2024.11.05 20:05:00.694800 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 15 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:02.768334 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:06.366846 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {62ec4990-945b-4d8e-9889-ca33c67862f8::202411_1_179_35}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:06.571074 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/ycz/nagdkjixnmacklliekxnsqpowgofe, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 71d1c71c-3197-4da3-99c0-a2d7f318faa9, upload has finished false.
2024.11.05 20:05:06.580849 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/ycz/nagdkjixnmacklliekxnsqpowgofe, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 71d1c71c-3197-4da3-99c0-a2d7f318faa9, upload has finished false
2024.11.05 20:05:06.617905 [ 3570 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/ycz/nagdkjixnmacklliekxnsqpowgofe, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 71d1c71c-3197-4da3-99c0-a2d7f318faa9, upload has finished false
2024.11.05 20:05:06.683025 [ 3570 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 18 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:10.149591 [ 3641 ] {} <Error> void DB::SystemLog<DB::PartLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::PartLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:10.665016 [ 3579 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key s3/ldf/catqctoqjmdnfoxyevqwmvyizngnd, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6731b877-9d32-4129-9a74-832e69eb3caf, upload has finished false.
2024.11.05 20:05:10.668378 [ 3579 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key s3/ldf/catqctoqjmdnfoxyevqwmvyizngnd, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6731b877-9d32-4129-9a74-832e69eb3caf, upload has finished false
2024.11.05 20:05:10.713855 [ 3579 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key s3/ldf/catqctoqjmdnfoxyevqwmvyizngnd, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 6731b877-9d32-4129-9a74-832e69eb3caf, upload has finished false
2024.11.05 20:05:10.916961 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column Referer): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_5_5_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 2 with max_rows_to_read = 6739): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:11.755009 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:15.454137 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/xku/uazhweotwlcvsfndmlzlbflljrqnn, total size 100663296, count 100663296, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 7223633d-d368-4789-8197-226e4b53fff3, upload has finished false.
2024.11.05 20:05:15.459936 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/xku/uazhweotwlcvsfndmlzlbflljrqnn, total size 100663296, count 100663296, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 7223633d-d368-4789-8197-226e4b53fff3, upload has finished false
2024.11.05 20:05:15.575583 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/xku/uazhweotwlcvsfndmlzlbflljrqnn, total size 100663296, count 100663296, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 7223633d-d368-4789-8197-226e4b53fff3, upload has finished false
2024.11.05 20:05:15.624957 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=2): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 25 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:16.136221 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column VisitID): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_5_5_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 1 with max_rows_to_read = 6739): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:19.076927 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column StartDate): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_2_2_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 0 with max_rows_to_read = 6592): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:20.009230 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 8 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:21.105380 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column VisitID): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_3_3_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 1 with max_rows_to_read = 6608): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:22.728995 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=2): (while reading column VisitID): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_1_1_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 0 with max_rows_to_read = 6658): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:23.616469 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 9 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:23.997721 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:29.305235 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/rce/zuovvnnojqqkkojkzovufkkrrlzxa, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id e75bb7e0-7b80-47d7-ba2a-b5a6ccdcf599, upload has finished false.
2024.11.05 20:05:29.306937 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/rce/zuovvnnojqqkkojkzovufkkrrlzxa, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id e75bb7e0-7b80-47d7-ba2a-b5a6ccdcf599, upload has finished false
2024.11.05 20:05:29.352599 [ 3571 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/rce/zuovvnnojqqkkojkzovufkkrrlzxa, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id e75bb7e0-7b80-47d7-ba2a-b5a6ccdcf599, upload has finished false
2024.11.05 20:05:29.620755 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 14 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:30.521609 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:31.410888 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:35.123732 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/jas/agppajjjyvwcyvzhqqwqtkfasxxxf, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 624b0270-fd00-4e6a-bc7d-9461f0b70c2b, upload has finished false.
2024.11.05 20:05:35.124582 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/jas/agppajjjyvwcyvzhqqwqtkfasxxxf, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 624b0270-fd00-4e6a-bc7d-9461f0b70c2b, upload has finished false
2024.11.05 20:05:35.149534 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/jas/agppajjjyvwcyvzhqqwqtkfasxxxf, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 624b0270-fd00-4e6a-bc7d-9461f0b70c2b, upload has finished false
2024.11.05 20:05:35.192376 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 14 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:36.047541 [ 5793 ] {6314755e-ab20-4780-8a07-d07eeb810fb4} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:41366) (comment: 02789_filesystem_cache_alignment.sh) (in query: SELECT count() FROM ( SELECT cache_path, file_size, tupleElement(file_segment_range, 2) - tupleElement(file_segment_range, 1) + 1 as file_segment_size, formatReadableSize(file_size) as formatted_file_size, formatReadableSize(file_segment_size) as formatted_file_segment_size, tupleElement(file_segment_range, 2) as end_offset FROM ( SELECT arrayJoin(cache_paths) AS cache_path, local_path, remote_path, size as file_size FROM system.remote_data_paths WHERE endsWith(local_path, '.bin') ) AS data_paths INNER JOIN system.filesystem_cache_log AS cache_log ON data_paths.remote_path = cache_log.source_file_path WHERE query_id = '22770' ) WHERE file_segment_size < file_size AND end_offset + 1 != file_size;), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 20:05:36.137905 [ 5793 ] {6314755e-ab20-4780-8a07-d07eeb810fb4} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::1]:9000 -> [::1]:41366). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/NetException.h:26: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x00000000134e9d48
6. /build/src/IO/WriteBufferFromPocoSocket.cpp:169: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000134e9244
7. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
8. /build/src/Server/TCPHandler.cpp:2253: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001d487855
9. /build/src/Server/TCPHandler.cpp:2330: DB::TCPHandler::sendLogs() @ 0x000000001d47c5ff
10. /build/src/Server/TCPHandler.cpp:693: DB::TCPHandler::runImpl() @ 0x000000001d471ac2
11. /build/src/Server/TCPHandler.cpp:2339: DB::TCPHandler::run() @ 0x000000001d4878c5
12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
17. ? @ 0x00007f3c91188ac3
18. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:36.146349 [ 5793 ] {6314755e-ab20-4780-8a07-d07eeb810fb4} <Error> TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=3, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Processors/Executors/PipelineExecutor.cpp:360: DB::PipelineExecutor::spawnThreads() @ 0x000000001d517507
10. /build/src/Processors/Executors/PipelineExecutor.cpp:404: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x000000001d515645
11. /build/src/Processors/Executors/PipelineExecutor.cpp:111: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001d514ab6
12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:83: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr<DB::ThreadGroup>, unsigned long, bool) @ 0x000000001d53a570
13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:110: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001d53a3df
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a355
15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001d53a339
16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001d53a29d
17. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001d53a1a7
18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d53a115
19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001d53a0f5
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d53a0d5
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d53a0a0
22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
24. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
25. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133ddb04
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133ddab5
27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(std::tuple<void, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x00000000133dda7d
28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000133dd882
29. ? @ 0x00007f3c91188ac3
30. ? @ 0x00007f3c9121a850

2024.11.05 20:05:36.153972 [ 5793 ] {6314755e-ab20-4780-8a07-d07eeb810fb4} <Warning> TCPHandler: Client has gone away.
2024.11.05 20:05:38.762951 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:44.329625 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=1): (while reading column VisitID): (while reading from part /var/lib/clickhouse/disks/s3_disk/store/f95/f9592c6f-8d15-4782-9f8a-9e1e27a04e21/201403_5_5_0/ in table test.visits (f9592c6f-8d15-4782-9f8a-9e1e27a04e21) located on disk s3_disk of type s3, from mark 19 with max_rows_to_read = 6739): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:47.160778 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:47.941478 [ 3679 ] {} <Error> void DB::SystemLog<DB::BlobStorageLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::BlobStorageLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:52.672635 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/fwg/iduuhfwkkcdyeqcjeqxhfuyxhofbe, total size 234881024, count 234881024, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 257a73b6-0fd2-4818-a17e-8e9d2f41e4ed, upload has finished false.
2024.11.05 20:05:52.674110 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/fwg/iduuhfwkkcdyeqcjeqxhfuyxhofbe, total size 234881024, count 234881024, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 257a73b6-0fd2-4818-a17e-8e9d2f41e4ed, upload has finished false
2024.11.05 20:05:52.752309 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/fwg/iduuhfwkkcdyeqcjeqxhfuyxhofbe, total size 234881024, count 234881024, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 257a73b6-0fd2-4818-a17e-8e9d2f41e4ed, upload has finished false
2024.11.05 20:05:52.844752 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 51 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:55.592899 [ 3673 ] {} <Error> void DB::SystemLog<DB::TransactionsInfoLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TransactionsInfoLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:147: DB::IMergeTreeDataPart::MinMaxIndex::store(std::vector<String, std::allocator<String>> const&, std::vector<std::shared_ptr<DB::IDataType const>, std::allocator<std::shared_ptr<DB::IDataType const>>> const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caad819
21. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:118: DB::IMergeTreeDataPart::MinMaxIndex::store(DB::MergeTreeData const&, DB::IDataPartStorage&, DB::MergeTreeDataPartChecksums&) const @ 0x000000001caac1fb
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:244: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a4cf
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
25. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
26. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
27. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
28. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:57.080775 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 8 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:05:57.379391 [ 3644 ] {} <Error> void DB::SystemLog<DB::QueryLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::QueryLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:02.080079 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/nod/gcjmqqqlcplqyspubfzvfeofndqvh, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 38172ea2-2c6c-4f88-ae3a-fef141b6e408, upload has finished false.
2024.11.05 20:06:02.081591 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/nod/gcjmqqqlcplqyspubfzvfeofndqvh, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 38172ea2-2c6c-4f88-ae3a-fef141b6e408, upload has finished false
2024.11.05 20:06:02.112044 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/nod/gcjmqqqlcplqyspubfzvfeofndqvh, total size 50331648, count 50331648, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 38172ea2-2c6c-4f88-ae3a-fef141b6e408, upload has finished false
2024.11.05 20:06:02.131855 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 13 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:08.219808 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/qdh/pjapjmcfemavheqqfihpcoznflpri, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id bda40f6a-0e6c-4bc1-aed3-3b24f249be21, upload has finished false.
2024.11.05 20:06:08.221106 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/qdh/pjapjmcfemavheqqfihpcoznflpri, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id bda40f6a-0e6c-4bc1-aed3-3b24f249be21, upload has finished false
2024.11.05 20:06:08.259434 [ 3569 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/qdh/pjapjmcfemavheqqfihpcoznflpri, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id bda40f6a-0e6c-4bc1-aed3-3b24f249be21, upload has finished false
2024.11.05 20:06:08.376988 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 15 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:10.738325 [ 3571 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 2 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:11.751073 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:13.414773 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:13.505791 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 7 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:14.047561 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:20.369696 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0aef53a4-984b-414b-bb72-6836bf38bb48::202411_178_187_2}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:20.866725 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0aef53a4-984b-414b-bb72-6836bf38bb48::202411_178_187_2}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:25.809383 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:296: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd3693
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:27.607075 [ 3640 ] {} <Error> void DB::SystemLog<DB::ZooKeeperLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ZooKeeperLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:274: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3abc4
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:36.383864 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/bnn/qdcwvefhgsduhqfvznpdyizmuadcr, total size 369098752, count 369098752, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 9b4ba6cf-d086-452b-a945-3641800a17de, upload has finished false.
2024.11.05 20:06:36.385570 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/bnn/qdcwvefhgsduhqfvznpdyizmuadcr, total size 369098752, count 369098752, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 9b4ba6cf-d086-452b-a945-3641800a17de, upload has finished false
2024.11.05 20:06:36.491870 [ 3582 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/bnn/qdcwvefhgsduhqfvznpdyizmuadcr, total size 369098752, count 369098752, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 9b4ba6cf-d086-452b-a945-3641800a17de, upload has finished false
2024.11.05 20:06:36.526033 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 77 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:37.367543 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=3). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:38.846610 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {4b81aa11-cc6c-476f-947b-cd29adf35c9b::202411_1_17_4}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:302: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b1cc
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:39.206426 [ 3569 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 4 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:40.224462 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=5). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:41.574015 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 2 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:43.445967 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_4_4_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 0 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:46.187995 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:48.760854 [ 3645 ] {} <Error> void DB::SystemLog<DB::OpenTelemetrySpanLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::OpenTelemetrySpanLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:51.662003 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:298: DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001ccd36af
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f1a
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:52.634637 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/hsk/kuellsgwoenzhphlxqszxfmltiily, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0a03607b-3fb5-4277-9288-847eb0855681, upload has finished false.
2024.11.05 20:06:52.635497 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/hsk/kuellsgwoenzhphlxqszxfmltiily, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0a03607b-3fb5-4277-9288-847eb0855681, upload has finished false
2024.11.05 20:06:52.688078 [ 3584 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/hsk/kuellsgwoenzhphlxqszxfmltiily, total size 117440512, count 117440512, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 0a03607b-3fb5-4277-9288-847eb0855681, upload has finished false
2024.11.05 20:06:52.723270 [ 3584 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_2_2_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 28 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:53.206524 [ 3654 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
24. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
25. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
26. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
27. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:55.311776 [ 3635 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TextLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:260: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3a8a0
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:06:57.395460 [ 3638 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:07:03.148087 [ 3572 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0aef53a4-984b-414b-bb72-6836bf38bb48::202411_178_192_3}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:07:07.675774 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:07:08.489407 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/nka/yzuhebmzvcgjqecgnmjomzqvyawgn, total size 218103808, count 218103808, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 439e8d77-fb1e-4026-809f-b6795c304465, upload has finished false.
2024.11.05 20:07:08.493011 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/nka/yzuhebmzvcgjqecgnmjomzqvyawgn, total size 218103808, count 218103808, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 439e8d77-fb1e-4026-809f-b6795c304465, upload has finished false
2024.11.05 20:07:08.584675 [ 3574 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/nka/yzuhebmzvcgjqecgnmjomzqvyawgn, total size 218103808, count 218103808, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 439e8d77-fb1e-4026-809f-b6795c304465, upload has finished false
2024.11.05 20:07:08.693100 [ 3574 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_5_5_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 48 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:07:09.466211 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {c04f5a1b-d832-4de4-96b0-dec4e591ea83::202411_1_163_61}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=2). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:442: DB::MergeTreeDataPartWriterOnDisk::fillPrimaryIndexChecksums(DB::MergeTreeDataPartChecksums&) @ 0x000000001cce40ba
21. /build/src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp:0: DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&, std::unordered_set<String, std::hash<String>, std::equal_to<String>, std::allocator<String>>&) @ 0x000000001ccd3f38
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:153: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf3883a
23. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
24. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
25. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
26. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
27. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
30. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:07:13.035500 [ 3576 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Error> virtual void DB::CachedOnDiskWriteBufferFromFile::nextImpl(): Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:567: DB::WriteBufferFromS3::writePart(DB::WriteBufferFromS3::PartData&&) @ 0x0000000019914d6e
18. /build/src/IO/WriteBufferFromS3.cpp:370: DB::WriteBufferFromS3::writeMultipartUpload() @ 0x0000000019908eb6
19. /build/src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::nextImpl() @ 0x000000001990857e
20. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
21. /build/src/Disks/IO/CachedOnDiskWriteBufferFromFile.cpp:236: DB::CachedOnDiskWriteBufferFromFile::nextImpl() @ 0x000000001a35e546
22. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
23. /build/src/IO/WriteBufferFromFileDecorator.cpp:71: DB::WriteBufferFromFileDecorator::nextImpl() @ 0x000000001a3593ad
24. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
25. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
26. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
27. /build/src/IO/WriteBuffer.h:76: DB::WriteBuffer::nextIfAtEnd() @ 0x000000000a931978
28. /build/src/IO/WriteBuffer.h:91: DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000a9317e0
29. /build/src/Compression/CompressedWriteBuffer.cpp:58: DB::CompressedWriteBuffer::nextImpl() @ 0x0000000019ddd7ed
30. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
31. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:07:13.505198 [ 3576 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key s3/jcg/zmsfhepkjexnjslmpauuiscaebknm, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 1bd36a16-f725-4e33-927c-741fee3b85b4, upload has finished false.
2024.11.05 20:07:13.512529 [ 3576 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key s3/jcg/zmsfhepkjexnjslmpauuiscaebknm, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 1bd36a16-f725-4e33-927c-741fee3b85b4, upload has finished false
2024.11.05 20:07:13.551595 [ 3576 ] {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key s3/jcg/zmsfhepkjexnjslmpauuiscaebknm, total size 83886080, count 83886080, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 1bd36a16-f725-4e33-927c-741fee3b85b4, upload has finished false
2024.11.05 20:07:13.889105 [ 3576 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {f9592c6f-8d15-4782-9f8a-9e1e27a04e21::201403_1_6_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:567: DB::WriteBufferFromS3::writePart(DB::WriteBufferFromS3::PartData&&) @ 0x0000000019914d6e
18. /build/src/IO/WriteBufferFromS3.cpp:370: DB::WriteBufferFromS3::writeMultipartUpload() @ 0x0000000019908eb6
19. /build/src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::nextImpl() @ 0x000000001990857e
20. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
21. /build/src/Disks/IO/CachedOnDiskWriteBufferFromFile.cpp:236: DB::CachedOnDiskWriteBufferFromFile::nextImpl() @ 0x000000001a35e546
22. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
23. /build/src/IO/WriteBufferFromFileDecorator.cpp:71: DB::WriteBufferFromFileDecorator::nextImpl() @ 0x000000001a3593ad
24. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
25. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
26. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
27. /build/src/IO/WriteBuffer.h:76: DB::WriteBuffer::nextIfAtEnd() @ 0x000000000a931978
28. /build/src/IO/WriteBuffer.h:91: DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000a9317e0
29. /build/src/Compression/CompressedWriteBuffer.cpp:58: DB::CompressedWriteBuffer::nextImpl() @ 0x0000000019ddd7ed
30. /build/src/IO/WriteBuffer.h:53: DB::WriteBuffer::next() @ 0x000000000a92b4ed
31. /build/src/IO/HashingWriteBuffer.h:64: DB::HashingWriteBuffer::nextImpl() @ 0x000000001ca7981d
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:07:14.834738 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/gib/oiqrtonxruixxfsjvnkvgzxjuhkpw, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 76a3799d-dee9-432d-8b81-a90b3321e140, upload has finished false.
2024.11.05 20:07:14.835896 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/gib/oiqrtonxruixxfsjvnkvgzxjuhkpw, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 76a3799d-dee9-432d-8b81-a90b3321e140, upload has finished false
2024.11.05 20:07:14.880976 [ 3581 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/gib/oiqrtonxruixxfsjvnkvgzxjuhkpw, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 76a3799d-dee9-432d-8b81-a90b3321e140, upload has finished false
2024.11.05 20:07:14.973778 [ 3581 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_1_1_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 16 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:07:15.270754 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {dbd666d3-2d23-40c6-8136-6ca2be336b4d::202411_1_154_29}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=4). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:323: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b644
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:07:18.055041 [ 3573 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0a233324-63e4-4351-8154-d23ef4927e18::202411_1_176_35}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce5722c
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57135
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce57063
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce56f15
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&&, unsigned long&&) @ 0x000000001ce544bb
24. /build/src/Storages/MergeTree/MergeTreeReaderCompact.cpp:37: DB::MergeTreeReaderCompact::MergeTreeReaderCompact(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::NamesAndTypesList, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*, DB::MarkCache*, DB::MarkRanges, DB::MergeTreeReaderSettings, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>>, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int) @ 0x000000001ce5270a
25. /build/src/Storages/MergeTree/MergeTreeReaderCompactSingleBuffer.h:16: DB::MergeTreeReaderCompactSingleBuffer::MergeTreeReaderCompactSingleBuffer<std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc2fa5
26. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::MergeTreeReaderCompactSingleBuffer>::__unique_single std::make_unique[abi:v15000]<DB::MergeTreeReaderCompactSingleBuffer, std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int>(std::shared_ptr<DB::LoadedMergeTreeDataPartInfoForReader>&, DB::NamesAndTypesList const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::UncompressedCache*&, DB::MarkCache*&, DB::MarkRanges const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int&&) @ 0x000000001ccc237d
27. /build/src/Storages/MergeTree/MergeTreeDataPartCompact.cpp:43: DB::MergeTreeDataPartCompact::getReader(DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::MarkRanges const&, std::unordered_map<String, DB::Field, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::Field>>> const&, DB::UncompressedCache*, DB::MarkCache*, std::shared_ptr<DB::AlterConversions const> const&, DB::MergeTreeReaderSettings const&, std::map<String, double, std::less<String>, std::allocator<std::pair<String const, double>>> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&) const @ 0x000000001ccbf6f3
28. /build/src/Storages/MergeTree/MergeTreeSequentialSource.cpp:177: DB::MergeTreeSequentialSource::MergeTreeSequentialSource(DB::MergeTreeSequentialSourceType, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>, std::vector<String, std::allocator<String>>, std::optional<DB::MarkRanges>, bool, bool, bool, bool) @ 0x000000001ce726f0
29. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeSequentialSource* std::construct_at[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, DB::MergeTreeSequentialSource*>(DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76847
30. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeSequentialSource>>::construct[abi:v15000]<DB::MergeTreeSequentialSource, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&, void, void>(std::allocator<DB::MergeTreeSequentialSource>&, DB::MergeTreeSequentialSource*, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce76504
31. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeSequentialSource, std::allocator<DB::MergeTreeSequentialSource>>::__shared_ptr_emplace[abi:v15000]<DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>, bool, bool&, bool&, bool&>(std::allocator<DB::MergeTreeSequentialSource>, DB::MergeTreeSequentialSourceType&, DB::MergeTreeData const&, std::shared_ptr<DB::StorageSnapshot> const&, std::shared_ptr<DB::IMergeTreeDataPart const>&, std::vector<String, std::allocator<String>>&, std::optional<DB::MarkRanges>&&, bool&&, bool&, bool&, bool&) @ 0x000000001ce761b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:07:21.655373 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/axh/chbddakohzlqucmsdntqgrtgiclca, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 380960c8-9e43-4171-a5e2-b51531186a58, upload has finished false.
2024.11.05 20:07:21.656184 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/axh/chbddakohzlqucmsdntqgrtgiclca, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 380960c8-9e43-4171-a5e2-b51531186a58, upload has finished false
2024.11.05 20:07:21.695023 [ 3575 ] {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/axh/chbddakohzlqucmsdntqgrtgiclca, total size 67108864, count 67108864, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 380960c8-9e43-4171-a5e2-b51531186a58, upload has finished false
2024.11.05 20:07:21.754255 [ 3575 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=74, jobs=0): (while reading column s): (while reading from part /var/lib/clickhouse/disks/s3/store/50a/50a7e901-55fe-42f8-9ca4-b0affd1a8885/all_3_3_0/ in table test_3.`02725_memory_for_merges` (50a7e901-55fe-42f8-9ca4-b0affd1a8885) located on disk s3 of type s3, from mark 16 with max_rows_to_read = 1): While executing MergeTreeSequentialSource. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority) @ 0x00000000190e9ae1
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000190e997b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000190e9930
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000190e98f0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000190e987b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000190eb5b1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000190e9590
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<DB::IAsynchronousReader::Result> DB::scheduleFromThreadPoolUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x00000000190e8e1c
17. /build/src/Disks/IO/ThreadPoolRemoteFSReader.cpp:109: DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request) @ 0x00000000190e863f
18. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:90: DB::AsynchronousBoundedReadBuffer::readAsync(char*, unsigned long, Priority) @ 0x0000000019365b12
19. /build/src/Disks/IO/AsynchronousBoundedReadBuffer.cpp:116: DB::AsynchronousBoundedReadBuffer::prefetch(Priority) @ 0x0000000019365e1e
20. /build/src/Compression/CompressedReadBufferFromFile.cpp:57: DB::CompressedReadBufferFromFile::prefetch(Priority) @ 0x0000000019ddcc4d
21. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:331: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6d968
22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6d7bd
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6d77d
24. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6d73d
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6d708
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
28. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
29. /build/src/DataTypes/Serializations/ISerialization.cpp:103: DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&) const @ 0x0000000019fd0c5c
30. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:322: DB::MergeTreeReaderWide::prefetchForColumn(Priority, DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, unsigned long, bool, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&) @ 0x000000001ce6b39b
31. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:103: DB::MergeTreeReaderWide::prefetchForAllColumns(Priority, unsigned long, unsigned long, unsigned long, bool) @ 0x000000001ce6b110
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:07:22.038402 [ 3582 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {dbd666d3-2d23-40c6-8136-6ca2be336b4d::202411_1_154_29}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:286: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3aefb
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:136: DB::MergedBlockOutputStream::finalizePart(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf38743
23. /build/src/Storages/MergeTree/MergeTask.cpp:815: DB::MergeTask::MergeProjectionsStage::finalizeProjectionsAndWholeMerge() const @ 0x000000001cb72abf
24. /build/src/Storages/MergeTree/MergeTask.cpp:840: DB::MergeTask::MergeProjectionsStage::execute() @ 0x000000001cb72dd8
25. /build/src/Storages/MergeTree/MergeTask.cpp:889: DB::MergeTask::execute() @ 0x000000001cb727b2
26. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:57: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f738
27. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
28. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
29. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
30. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
31. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:07:23.309778 [ 3583 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {50a7e901-55fe-42f8-9ca4-b0affd1a8885::all_1_5_1}: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=50, jobs=12). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<std::shared_ptr<DB::MarksInCompressedFile>> (DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority)::operator()(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, Priority) @ 0x000000001ce20491
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce2033b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001ce202f0
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce202b0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce2023b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001ce21ed1
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001ce1ff50
16. /build/src/Common/threadPoolCallbackRunner.h:70: std::future<std::shared_ptr<DB::MarksInCompressedFile>> DB::scheduleFromThreadPoolUnsafe<std::shared_ptr<DB::MarksInCompressedFile>, DB::MergeTreeMarksLoader::loadMarksAsync()::$_0>(DB::MergeTreeMarksLoader::loadMarksAsync()::$_0&&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&, Priority) @ 0x000000001ce1f89c
17. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:234: DB::MergeTreeMarksLoader::loadMarksAsync() @ 0x000000001ce1dec3
18. /build/src/Storages/MergeTree/MergeTreeMarksLoader.cpp:69: DB::MergeTreeMarksLoader::MergeTreeMarksLoader(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>, DB::MarkCache*, String const&, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool, DB::ReadSettings const&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*, unsigned long) @ 0x000000001ce1ddb3
19. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::MergeTreeMarksLoader* std::construct_at[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, DB::MergeTreeMarksLoader*>(DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f6ac
20. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::MergeTreeMarksLoader>>::construct[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void, void>(std::allocator<DB::MergeTreeMarksLoader>&, DB::MergeTreeMarksLoader*, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f5b5
21. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>>::__shared_ptr_emplace[abi:v15000]<std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int>(std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f4e3
22. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::MergeTreeMarksLoader> std::allocate_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::allocator<DB::MergeTreeMarksLoader>, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::allocator<DB::MergeTreeMarksLoader> const&, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6f395
23. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::MergeTreeMarksLoader> std::make_shared[abi:v15000]<DB::MergeTreeMarksLoader, std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String, unsigned long, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int, void>(std::shared_ptr<DB::IMergeTreeDataPartInfoForReader>&, DB::MarkCache* const&, String&&, unsigned long&&, DB::MergeTreeIndexGranularityInfo const&, bool&, DB::ReadSettings&, ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>*&, int&&) @ 0x000000001ce6ec9b
24. /build/src/Storages/MergeTree/MergeTreeReaderWide.cpp:231: DB::MergeTreeReaderWide::addStreams(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::$_0::operator()(DB::ISerialization::SubstreamPath const&) const @ 0x000000001ce6c6d3
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001ce6c3dd
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001ce6c39d
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001ce6c35d
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001ce6c328
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000019fd727e
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000019fd36fd
31. /build/src/DataTypes/Serializations/ISerialization.cpp:93: DB::ISerialization::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x0000000019fd0ad6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:07:23.547264 [ 3674 ] {} <Error> void DB::SystemLog<DB::ProcessorProfileLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::ProcessorProfileLogElement]: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=33, jobs=1). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x00000000133dd268
6. /build/src/Common/ThreadPool.cpp:203: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x00000000133e222d
7. /build/src/Common/ThreadPool.cpp:219: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x00000000133d7418
8. /build/src/Common/ThreadPool.cpp:326: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x00000000133d729c
9. /build/src/Common/threadPoolCallbackRunner.h:60: std::function<std::future<void> (std::function<void ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<void, std::function<void ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, String const&)::'lambda'(std::function<void ()>&&, Priority)::operator()(std::function<void ()>&&, Priority) @ 0x00000000198cb8b6
10. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000198cb75b
11. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x00000000198cb710
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000198cb6d0
13. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000198cb67b
14. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013398051
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000013393770
16. /build/src/Common/ThreadPoolTaskTracker.cpp:130: DB::TaskTracker::add(std::function<void ()>&&) @ 0x0000000019372143
17. /build/src/IO/WriteBufferFromS3.cpp:739: DB::WriteBufferFromS3::makeSinglepartUpload(DB::WriteBufferFromS3::PartData&&) @ 0x000000001990a1ab
18. /build/src/IO/WriteBufferFromS3.cpp:180: DB::WriteBufferFromS3::preFinalize() @ 0x000000001990990f
19. /build/src/IO/WriteBufferFromFileDecorator.cpp:62: DB::WriteBufferFromFileDecorator::preFinalize() @ 0x000000001a35911e
20. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:294: DB::MergedBlockOutputStream::finalizePartOnDisk(std::shared_ptr<DB::IMergeTreeDataPart> const&, DB::MergeTreeDataPartChecksums&) @ 0x000000001cf3b0b6
21. /build/src/Storages/MergeTree/MergedBlockOutputStream.cpp:180: DB::MergedBlockOutputStream::finalizePartAsync(std::shared_ptr<DB::IMergeTreeDataPart> const&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x000000001cf39799
22. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:628: DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, long, bool) @ 0x000000001cf87ab3
23. /build/src/Storages/MergeTree/MergeTreeDataWriter.cpp:404: DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>) @ 0x000000001cf83e5d
24. /build/src/Storages/MergeTree/MergeTreeSink.cpp:86: DB::MergeTreeSink::consume(DB::Chunk) @ 0x000000001d13c58f
25. /build/src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001dbbde25
26. /build/src/Processors/Transforms/ExceptionKeepingTransform.cpp:150: DB::ExceptionKeepingTransform::work()::$_1::operator()() const @ 0x000000001da423fc
27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001da423b5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001da42395
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001da42375
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001da4233d
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:07:33.939652 [ 3576 ] {} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/yke/oydqdvsafyundskjuzugnwjgpzdes, total size 369098752, count 369098752, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8916fdaf-5995-4787-9686-739f59a6dbec, upload has finished false.
2024.11.05 20:07:33.939786 [ 3576 ] {} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/yke/oydqdvsafyundskjuzugnwjgpzdes, total size 369098752, count 369098752, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8916fdaf-5995-4787-9686-739f59a6dbec, upload has finished false
2024.11.05 20:07:34.039453 [ 3576 ] {} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/yke/oydqdvsafyundskjuzugnwjgpzdes, total size 369098752, count 369098752, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 8916fdaf-5995-4787-9686-739f59a6dbec, upload has finished false
2024.11.05 20:08:18.625938 [ 3579 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {0aef53a4-984b-414b-bb72-6836bf38bb48::202411_1_197_45}: Code: 241. DB::Exception: Memory tracker (for thread): fault injected (at specific point). (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*, char const*>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type>, char const*&&, char const*&&) @ 0x00000000132eb36c
6. /build/src/Common/MemoryTracker.cpp:184: MemoryTracker::injectFault() const @ 0x00000000132e875f
7. /build/src/Common/CurrentMemoryTracker.cpp:129: CurrentMemoryTracker::injectFault() @ 0x0000000013257305
8. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:149: DB::MergePlainMergeTreeTask::finish() @ 0x000000001d140118
9. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:71: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f876
10. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
11. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
12. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001cbaa415
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001cbaa3dd
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:08:46.685878 [ 3577 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {c04f5a1b-d832-4de4-96b0-dec4e591ea83::202411_1_173_71}: Code: 241. DB::Exception: Memory tracker (for thread): fault injected (at specific point). (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*, char const*>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type>, char const*&&, char const*&&) @ 0x00000000132eb36c
6. /build/src/Common/MemoryTracker.cpp:184: MemoryTracker::injectFault() const @ 0x00000000132e875f
7. /build/src/Common/CurrentMemoryTracker.cpp:129: CurrentMemoryTracker::injectFault() @ 0x0000000013257305
8. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:149: DB::MergePlainMergeTreeTask::finish() @ 0x000000001d140118
9. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:71: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f876
10. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
11. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
12. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001cbaa415
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001cbaa3dd
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:11:24.363525 [ 3578 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {29fdad69-1244-4c25-af3c-70c43e910f15::202411_202_363_54}: Code: 241. DB::Exception: Memory tracker (for thread): fault injected (at specific point). (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<char const*, char const*>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type>, char const*&&, char const*&&) @ 0x00000000132eb36c
6. /build/src/Common/MemoryTracker.cpp:184: MemoryTracker::injectFault() const @ 0x00000000132e875f
7. /build/src/Common/CurrentMemoryTracker.cpp:129: CurrentMemoryTracker::injectFault() @ 0x0000000013257305
8. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:149: DB::MergePlainMergeTreeTask::finish() @ 0x000000001d140118
9. /build/src/Storages/MergeTree/MergePlainMergeTreeTask.cpp:71: DB::MergePlainMergeTreeTask::executeStep() @ 0x000000001d13f876
10. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:282: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::routine(std::shared_ptr<DB::TaskRuntimeData>) @ 0x000000001cba1ded
11. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:348: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x000000001cba290e
12. /build/src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57: DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::MergeTreeBackgroundExecutor(String, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::basic_string_view<char, std::char_traits<char>>)::'lambda'()::operator()() const @ 0x000000001cbaa478
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001cbaa455
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001cbaa435
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001cbaa415
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001cbaa3dd
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
19. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
20. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
24. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
31. /build/src/Common/ThreadPool.cpp:536: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x00000000133d60b1
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:12:36.693532 [ 5215 ] {e1b73ff3-b6f5-4635-bf2c-0d768dbc789b} <Error> executeQuery: Code: 521. DB::ErrnoException: Cannot rename /var/lib/clickhouse/metadata/test.sql.tmp to /var/lib/clickhouse/metadata/test.sql because the second path already exists: , errno: 17, strerror: File exists. (ATOMIC_RENAME_FAIL) (version 24.3.12.76.altinitystable (altinity build)) (from [::1]:40300) (in query: CREATE DATABASE IF NOT EXISTS test ), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:194: DB::ErrnoException::ErrnoException<String const&, String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String const&>::type>, String const&, String const&) @ 0x000000001a0d58cc
6. /build/src/Common/atomicRename.cpp:90: DB::renameat2(String const&, String const&, int) @ 0x000000001a0d5312
7. /build/src/Common/atomicRename.cpp:211: DB::renameNoReplace(String const&, String const&) @ 0x000000001a0d5102
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:330: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00b8b0
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
10. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
11. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
12. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
13. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
14. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850

2024.11.05 20:12:36.694786 [ 5215 ] {e1b73ff3-b6f5-4635-bf2c-0d768dbc789b} <Error> DynamicQueryHandler: Code: 521. DB::ErrnoException: Cannot rename /var/lib/clickhouse/metadata/test.sql.tmp to /var/lib/clickhouse/metadata/test.sql because the second path already exists: , errno: 17, strerror: File exists. (ATOMIC_RENAME_FAIL), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:194: DB::ErrnoException::ErrnoException<String const&, String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String const&>::type>, String const&, String const&) @ 0x000000001a0d58cc
6. /build/src/Common/atomicRename.cpp:90: DB::renameat2(String const&, String const&, int) @ 0x000000001a0d5312
7. /build/src/Common/atomicRename.cpp:211: DB::renameNoReplace(String const&, String const&) @ 0x000000001a0d5102
8. /build/src/Interpreters/InterpreterCreateQuery.cpp:330: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000001b00b8b0
9. /build/src/Interpreters/InterpreterCreateQuery.cpp:1791: DB::InterpreterCreateQuery::execute() @ 0x000000001b02183f
10. /build/src/Interpreters/executeQuery.cpp:1195: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001bbb2ed7
11. /build/src/Interpreters/executeQuery.cpp:1496: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001bbb5689
12. /build/src/Server/HTTPHandler.cpp:891: DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3dac89
13. /build/src/Server/HTTPHandler.cpp:1090: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x000000001d3ddd70
14. /build/src/Server/HTTP/HTTPServerConnection.cpp:70: DB::HTTPServerConnection::run() @ 0x000000001d4967c7
15. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000228b3619
16. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000228b3e68
17. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x0000000022a8b194
18. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x0000000022a87f3a
19. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000022a86c7e
20. ? @ 0x00007f3c91188ac3
21. ? @ 0x00007f3c9121a850
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:13:05.418718 [ 2811 ] {} <Warning> Application: Closed all listening sockets. Waiting for 3 outstanding connections.
2024.11.05 20:13:10.323824 [ 2811 ] {} <Warning> Application: Closed connections. But 3 remain. Tip: To increase wait time add to config: <shutdown_wait_unfinished>60</shutdown_wait_unfinished>
2024.11.05 20:13:10.324040 [ 2811 ] {} <Warning> Application: Will shutdown forcefully.
2024.11.05 20:13:12.377817 [ 9011 ] {} <Warning> Context: Server was built in debug mode. It will work slowly.
2024.11.05 20:13:12.409211 [ 9011 ] {} <Warning> Context: Server logging level is set to 'test' and performance is degraded. This cannot be used in production.
2024.11.05 20:13:12.410758 [ 9011 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
2024.11.05 20:13:12.413489 [ 9011 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.11.05 20:13:17.763040 [ 9011 ] {} <Warning> CloudPlacementInfo: Placement info has not been loaded
2024.11.05 20:13:18.182694 [ 9011 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.11.05 20:13:18.188529 [ 9011 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.11.05 20:13:18.188626 [ 9011 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.11.05 20:13:18.240286 [ 9665 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.11.05 20:13:22.911976 [ 9011 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.11.05 20:13:23.419332 [ 9011 ] {} <Warning> loadMetadata: Removing temporary file /var/lib/clickhouse/metadata/test.sql.tmp
2024.11.05 20:13:23.622905 [ 9011 ] {} <Warning> DatabaseReplicated (test_k6cbek3q): Database test_k6cbek3q exists locally, but marked dropped in ZooKeeper (/test/db/test_8/replicas/s1|r1). Will not try to start it up
2024.11.05 20:13:23.646286 [ 9011 ] {} <Warning> TablesLoader: Tables test_15.discounts_dict depend on default.discounts, but seems like that does not exist. Will ignore it and try to load existing tables
2024.11.05 20:13:25.751920 [ 9831 ] {} <Error> void DB::AsyncLoader::worker(Pool &): Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:13:32.218460 [ 9703 ] {} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/pte/wetpbychdyixcryrytnavoglharia, total size 419430400, count 419430400, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 4bc05451-5b8a-450a-834a-7e76073ded66, upload has finished false.
2024.11.05 20:13:32.218696 [ 9703 ] {} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/pte/wetpbychdyixcryrytnavoglharia, total size 419430400, count 419430400, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 4bc05451-5b8a-450a-834a-7e76073ded66, upload has finished false
2024.11.05 20:13:32.279233 [ 9703 ] {} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/pte/wetpbychdyixcryrytnavoglharia, total size 419430400, count 419430400, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 4bc05451-5b8a-450a-834a-7e76073ded66, upload has finished false
2024.11.05 20:13:35.197803 [ 9668 ] {} <Error> RaftInstance: failed to accept a rpc connection due to error 125, Operation canceled
2024.11.05 20:13:36.836641 [ 9011 ] {} <Error> Application: Code: 696. DB::Exception: Load job 'startup table test_15.wv' -> Code: 695. DB::Exception: Load job 'load table test_15.wv' failed: Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED) (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_CANCELED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:73: DB::cancelOnDependencyFailure(std::shared_ptr<DB::LoadJob> const&, std::shared_ptr<DB::LoadJob> const&, std::exception_ptr&) @ 0x00000000135e7a76
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0d3bed
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0d3bad
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0d3b6d
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0d3b15
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013603b4e
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f708d
13. /build/src/Common/AsyncLoader.cpp:627: DB::AsyncLoader::finish(std::shared_ptr<DB::LoadJob> const&, DB::LoadStatus, std::exception_ptr, std::unique_lock<std::mutex>&) @ 0x00000000135eea1f
14. /build/src/Common/AsyncLoader.cpp:898: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f321e
15. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
22. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
23. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
25. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
26. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
27. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:13:39.142675 [ 9907 ] {} <Warning> Context: Server was built in debug mode. It will work slowly.
2024.11.05 20:13:39.176028 [ 9907 ] {} <Warning> Context: Server logging level is set to 'test' and performance is degraded. This cannot be used in production.
2024.11.05 20:13:39.177540 [ 9907 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
2024.11.05 20:13:39.180498 [ 9907 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.11.05 20:13:46.895922 [ 9907 ] {} <Warning> CloudPlacementInfo: Placement info has not been loaded
2024.11.05 20:13:47.330110 [ 9907 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.11.05 20:13:47.336458 [ 9907 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.11.05 20:13:47.336594 [ 9907 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.11.05 20:13:47.388614 [ 10567 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.11.05 20:13:52.276255 [ 9907 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.11.05 20:13:53.732566 [ 9907 ] {} <Warning> DatabaseReplicated (test_k6cbek3q): Database test_k6cbek3q exists locally, but marked dropped in ZooKeeper (/test/db/test_8/replicas/s1|r1). Will not try to start it up
2024.11.05 20:13:53.790553 [ 9907 ] {} <Warning> TablesLoader: Tables test_15.discounts_dict depend on default.discounts, but seems like that does not exist. Will ignore it and try to load existing tables
2024.11.05 20:13:55.504979 [ 10781 ] {} <Error> void DB::AsyncLoader::worker(Pool &): Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:14:03.926466 [ 10570 ] {} <Error> RaftInstance: failed to accept a rpc connection due to error 125, Operation canceled
2024.11.05 20:14:05.531346 [ 9907 ] {} <Error> Application: Code: 696. DB::Exception: Load job 'startup table test_15.wv' -> Code: 695. DB::Exception: Load job 'load table test_15.wv' failed: Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED) (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_CANCELED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:73: DB::cancelOnDependencyFailure(std::shared_ptr<DB::LoadJob> const&, std::shared_ptr<DB::LoadJob> const&, std::exception_ptr&) @ 0x00000000135e7a76
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0d3bed
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0d3bad
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0d3b6d
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0d3b15
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013603b4e
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f708d
13. /build/src/Common/AsyncLoader.cpp:627: DB::AsyncLoader::finish(std::shared_ptr<DB::LoadJob> const&, DB::LoadStatus, std::exception_ptr, std::unique_lock<std::mutex>&) @ 0x00000000135eea1f
14. /build/src/Common/AsyncLoader.cpp:898: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f321e
15. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
22. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
23. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
25. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
26. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
27. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:14:06.836391 [ 10802 ] {} <Warning> Context: Server was built in debug mode. It will work slowly.
2024.11.05 20:14:06.871833 [ 10802 ] {} <Warning> Context: Server logging level is set to 'test' and performance is degraded. This cannot be used in production.
2024.11.05 20:14:06.873218 [ 10802 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
2024.11.05 20:14:06.875901 [ 10802 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.11.05 20:14:15.293333 [ 10802 ] {} <Warning> CloudPlacementInfo: Placement info has not been loaded
2024.11.05 20:14:15.717752 [ 10802 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.11.05 20:14:15.723680 [ 10802 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.11.05 20:14:15.723777 [ 10802 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.11.05 20:14:15.775406 [ 11465 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.11.05 20:14:20.302839 [ 10802 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.11.05 20:14:21.613281 [ 10802 ] {} <Warning> DatabaseReplicated (test_k6cbek3q): Database test_k6cbek3q exists locally, but marked dropped in ZooKeeper (/test/db/test_8/replicas/s1|r1). Will not try to start it up
2024.11.05 20:14:21.645750 [ 10802 ] {} <Warning> TablesLoader: Tables test_15.discounts_dict depend on default.discounts, but seems like that does not exist. Will ignore it and try to load existing tables
2024.11.05 20:14:22.230198 [ 11577 ] {} <Error> void DB::AsyncLoader::worker(Pool &): Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:14:33.234331 [ 11468 ] {} <Error> RaftInstance: failed to accept a rpc connection due to error 125, Operation canceled
2024.11.05 20:14:34.774244 [ 10802 ] {} <Error> Application: Code: 696. DB::Exception: Load job 'startup table test_15.wv' -> Code: 695. DB::Exception: Load job 'load table test_15.wv' failed: Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED) (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_CANCELED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:73: DB::cancelOnDependencyFailure(std::shared_ptr<DB::LoadJob> const&, std::shared_ptr<DB::LoadJob> const&, std::exception_ptr&) @ 0x00000000135e7a76
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0d3bed
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0d3bad
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0d3b6d
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0d3b15
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013603b4e
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f708d
13. /build/src/Common/AsyncLoader.cpp:627: DB::AsyncLoader::finish(std::shared_ptr<DB::LoadJob> const&, DB::LoadStatus, std::exception_ptr, std::unique_lock<std::mutex>&) @ 0x00000000135eea1f
14. /build/src/Common/AsyncLoader.cpp:898: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f321e
15. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
22. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
23. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
25. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
26. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
27. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:14:36.087541 [ 11685 ] {} <Warning> Context: Server was built in debug mode. It will work slowly.
2024.11.05 20:14:36.118777 [ 11685 ] {} <Warning> Context: Server logging level is set to 'test' and performance is degraded. This cannot be used in production.
2024.11.05 20:14:36.120138 [ 11685 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
2024.11.05 20:14:36.122818 [ 11685 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.11.05 20:14:44.314327 [ 11685 ] {} <Warning> CloudPlacementInfo: Placement info has not been loaded
2024.11.05 20:14:44.739045 [ 11685 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.11.05 20:14:44.746253 [ 11685 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.11.05 20:14:44.746369 [ 11685 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.11.05 20:14:44.798125 [ 12346 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.11.05 20:14:49.693481 [ 11685 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.11.05 20:14:51.183828 [ 11685 ] {} <Warning> DatabaseReplicated (test_k6cbek3q): Database test_k6cbek3q exists locally, but marked dropped in ZooKeeper (/test/db/test_8/replicas/s1|r1). Will not try to start it up
2024.11.05 20:14:51.216325 [ 11685 ] {} <Warning> TablesLoader: Tables test_15.discounts_dict depend on default.discounts, but seems like that does not exist. Will ignore it and try to load existing tables
2024.11.05 20:14:52.705419 [ 12531 ] {} <Error> void DB::AsyncLoader::worker(Pool &): Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:14:57.243034 [ 12391 ] {} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/enu/snsmdlpvqoutdrglxkousniuvbgmw, total size 335544320, count 335544320, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id a71307cb-e053-468d-998b-28ad2a36c443, upload has finished false.
2024.11.05 20:14:57.243275 [ 12391 ] {} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/enu/snsmdlpvqoutdrglxkousniuvbgmw, total size 335544320, count 335544320, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id a71307cb-e053-468d-998b-28ad2a36c443, upload has finished false
2024.11.05 20:14:57.295602 [ 12391 ] {} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/enu/snsmdlpvqoutdrglxkousniuvbgmw, total size 335544320, count 335544320, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id a71307cb-e053-468d-998b-28ad2a36c443, upload has finished false
2024.11.05 20:15:02.023577 [ 11685 ] {} <Error> Application: Code: 696. DB::Exception: Load job 'startup table test_15.wv' -> Code: 695. DB::Exception: Load job 'load table test_15.wv' failed: Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED) (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_CANCELED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:73: DB::cancelOnDependencyFailure(std::shared_ptr<DB::LoadJob> const&, std::shared_ptr<DB::LoadJob> const&, std::exception_ptr&) @ 0x00000000135e7a76
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0d3bed
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0d3bad
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0d3b6d
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0d3b15
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013603b4e
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f708d
13. /build/src/Common/AsyncLoader.cpp:627: DB::AsyncLoader::finish(std::shared_ptr<DB::LoadJob> const&, DB::LoadStatus, std::exception_ptr, std::unique_lock<std::mutex>&) @ 0x00000000135eea1f
14. /build/src/Common/AsyncLoader.cpp:898: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f321e
15. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
22. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
23. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
25. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
26. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
27. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:15:03.335137 [ 12561 ] {} <Warning> Context: Server was built in debug mode. It will work slowly.
2024.11.05 20:15:03.366827 [ 12561 ] {} <Warning> Context: Server logging level is set to 'test' and performance is degraded. This cannot be used in production.
2024.11.05 20:15:03.368276 [ 12561 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
2024.11.05 20:15:03.371011 [ 12561 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.11.05 20:15:11.593889 [ 12561 ] {} <Warning> CloudPlacementInfo: Placement info has not been loaded
2024.11.05 20:15:12.047724 [ 12561 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.11.05 20:15:12.053641 [ 12561 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.11.05 20:15:12.053740 [ 12561 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.11.05 20:15:12.105377 [ 13223 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.11.05 20:15:17.017901 [ 12561 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.11.05 20:15:18.339097 [ 12561 ] {} <Warning> DatabaseReplicated (test_k6cbek3q): Database test_k6cbek3q exists locally, but marked dropped in ZooKeeper (/test/db/test_8/replicas/s1|r1). Will not try to start it up
2024.11.05 20:15:18.370776 [ 12561 ] {} <Warning> TablesLoader: Tables test_15.discounts_dict depend on default.discounts, but seems like that does not exist. Will ignore it and try to load existing tables
2024.11.05 20:15:20.069419 [ 13322 ] {} <Error> void DB::AsyncLoader::worker(Pool &): Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:15:27.666296 [ 13226 ] {} <Error> RaftInstance: failed to accept a rpc connection due to error 125, Operation canceled
2024.11.05 20:15:29.211200 [ 12561 ] {} <Error> Application: Code: 696. DB::Exception: Load job 'startup table test_15.wv' -> Code: 695. DB::Exception: Load job 'load table test_15.wv' failed: Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED) (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_CANCELED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:73: DB::cancelOnDependencyFailure(std::shared_ptr<DB::LoadJob> const&, std::shared_ptr<DB::LoadJob> const&, std::exception_ptr&) @ 0x00000000135e7a76
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0d3bed
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0d3bad
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0d3b6d
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0d3b15
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013603b4e
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f708d
13. /build/src/Common/AsyncLoader.cpp:627: DB::AsyncLoader::finish(std::shared_ptr<DB::LoadJob> const&, DB::LoadStatus, std::exception_ptr, std::unique_lock<std::mutex>&) @ 0x00000000135eea1f
14. /build/src/Common/AsyncLoader.cpp:898: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f321e
15. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
22. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
23. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
25. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
26. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
27. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:15:30.504562 [ 13446 ] {} <Warning> Context: Server was built in debug mode. It will work slowly.
2024.11.05 20:15:30.536230 [ 13446 ] {} <Warning> Context: Server logging level is set to 'test' and performance is degraded. This cannot be used in production.
2024.11.05 20:15:30.537838 [ 13446 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
2024.11.05 20:15:30.540604 [ 13446 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.11.05 20:15:38.692495 [ 13446 ] {} <Warning> CloudPlacementInfo: Placement info has not been loaded
2024.11.05 20:15:39.123363 [ 13446 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.11.05 20:15:39.129320 [ 13446 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.11.05 20:15:39.129417 [ 13446 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.11.05 20:15:39.180986 [ 14107 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.11.05 20:15:44.214758 [ 13446 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.11.05 20:15:45.293267 [ 13446 ] {} <Warning> DatabaseReplicated (test_k6cbek3q): Database test_k6cbek3q exists locally, but marked dropped in ZooKeeper (/test/db/test_8/replicas/s1|r1). Will not try to start it up
2024.11.05 20:15:45.335555 [ 13446 ] {} <Warning> TablesLoader: Tables test_15.discounts_dict depend on default.discounts, but seems like that does not exist. Will ignore it and try to load existing tables
2024.11.05 20:15:47.111008 [ 14275 ] {} <Error> void DB::AsyncLoader::worker(Pool &): Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:15:54.310600 [ 14110 ] {} <Error> RaftInstance: failed to accept a rpc connection due to error 125, Operation canceled
2024.11.05 20:15:55.738410 [ 13446 ] {} <Error> Application: Code: 696. DB::Exception: Load job 'startup table test_15.wv' -> Code: 695. DB::Exception: Load job 'load table test_15.wv' failed: Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED) (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_CANCELED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:73: DB::cancelOnDependencyFailure(std::shared_ptr<DB::LoadJob> const&, std::shared_ptr<DB::LoadJob> const&, std::exception_ptr&) @ 0x00000000135e7a76
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0d3bed
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0d3bad
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0d3b6d
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0d3b15
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013603b4e
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f708d
13. /build/src/Common/AsyncLoader.cpp:627: DB::AsyncLoader::finish(std::shared_ptr<DB::LoadJob> const&, DB::LoadStatus, std::exception_ptr, std::unique_lock<std::mutex>&) @ 0x00000000135eea1f
14. /build/src/Common/AsyncLoader.cpp:898: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f321e
15. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
22. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
23. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
25. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
26. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
27. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:15:57.043799 [ 14323 ] {} <Warning> Context: Server was built in debug mode. It will work slowly.
2024.11.05 20:15:57.075557 [ 14323 ] {} <Warning> Context: Server logging level is set to 'test' and performance is degraded. This cannot be used in production.
2024.11.05 20:15:57.076986 [ 14323 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
2024.11.05 20:15:57.079905 [ 14323 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.11.05 20:16:04.914468 [ 14323 ] {} <Warning> CloudPlacementInfo: Placement info has not been loaded
2024.11.05 20:16:05.363553 [ 14323 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.11.05 20:16:05.369452 [ 14323 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.11.05 20:16:05.369565 [ 14323 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.11.05 20:16:05.421309 [ 14983 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.11.05 20:16:10.445189 [ 14323 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.11.05 20:16:11.550015 [ 14323 ] {} <Warning> DatabaseReplicated (test_k6cbek3q): Database test_k6cbek3q exists locally, but marked dropped in ZooKeeper (/test/db/test_8/replicas/s1|r1). Will not try to start it up
2024.11.05 20:16:11.589657 [ 14323 ] {} <Warning> TablesLoader: Tables test_15.discounts_dict depend on default.discounts, but seems like that does not exist. Will ignore it and try to load existing tables
2024.11.05 20:16:12.349067 [ 15146 ] {} <Error> void DB::AsyncLoader::worker(Pool &): Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:16:17.868880 [ 15021 ] {} <Warning> WriteBufferFromS3: WriteBufferFromS3 was neither finished nor aborted, try to abort upload in destructor. Details: bucket test, key test/xvf/kjwmclihmfxbasmxdyhfinolhricv, total size 352321536, count 352321536, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 45656047-df57-4541-8593-3c41a39e7dd6, upload has finished false.
2024.11.05 20:16:17.869117 [ 15021 ] {} <Warning> WriteBufferFromS3: Abort multipart upload. Details: bucket test, key test/xvf/kjwmclihmfxbasmxdyhfinolhricv, total size 352321536, count 352321536, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 45656047-df57-4541-8593-3c41a39e7dd6, upload has finished false
2024.11.05 20:16:17.921214 [ 15021 ] {} <Warning> WriteBufferFromS3: Multipart upload has aborted successfully. Details: bucket test, key test/xvf/kjwmclihmfxbasmxdyhfinolhricv, total size 352321536, count 352321536, hidden_size 0, offset 0, with pool: true, prefinalized false, finalized false, upload id 45656047-df57-4541-8593-3c41a39e7dd6, upload has finished false
2024.11.05 20:16:22.459389 [ 14323 ] {} <Error> Application: Code: 696. DB::Exception: Load job 'startup table test_15.wv' -> Code: 695. DB::Exception: Load job 'load table test_15.wv' failed: Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED) (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_CANCELED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:73: DB::cancelOnDependencyFailure(std::shared_ptr<DB::LoadJob> const&, std::shared_ptr<DB::LoadJob> const&, std::exception_ptr&) @ 0x00000000135e7a76
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0d3bed
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0d3bad
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0d3b6d
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0d3b15
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013603b4e
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f708d
13. /build/src/Common/AsyncLoader.cpp:627: DB::AsyncLoader::finish(std::shared_ptr<DB::LoadJob> const&, DB::LoadStatus, std::exception_ptr, std::unique_lock<std::mutex>&) @ 0x00000000135eea1f
14. /build/src/Common/AsyncLoader.cpp:898: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f321e
15. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
22. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
23. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
25. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
26. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
27. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:16:23.835251 [ 15195 ] {} <Warning> Context: Server was built in debug mode. It will work slowly.
2024.11.05 20:16:23.878512 [ 15195 ] {} <Warning> Context: Server logging level is set to 'test' and performance is degraded. This cannot be used in production.
2024.11.05 20:16:23.880413 [ 15195 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
2024.11.05 20:16:23.884037 [ 15195 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.11.05 20:16:31.612594 [ 15195 ] {} <Warning> CloudPlacementInfo: Placement info has not been loaded
2024.11.05 20:16:32.066952 [ 15195 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.11.05 20:16:32.073695 [ 15195 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.11.05 20:16:32.073807 [ 15195 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.11.05 20:16:32.125497 [ 15855 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.11.05 20:16:37.123254 [ 15195 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.11.05 20:16:38.430593 [ 15195 ] {} <Warning> DatabaseReplicated (test_k6cbek3q): Database test_k6cbek3q exists locally, but marked dropped in ZooKeeper (/test/db/test_8/replicas/s1|r1). Will not try to start it up
2024.11.05 20:16:38.460400 [ 15195 ] {} <Warning> TablesLoader: Tables test_15.discounts_dict depend on default.discounts, but seems like that does not exist. Will ignore it and try to load existing tables
2024.11.05 20:16:39.878169 [ 16028 ] {} <Error> void DB::AsyncLoader::worker(Pool &): Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:16:51.116355 [ 15195 ] {} <Error> Application: Code: 696. DB::Exception: Load job 'startup table test_15.wv' -> Code: 695. DB::Exception: Load job 'load table test_15.wv' failed: Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED) (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_CANCELED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:73: DB::cancelOnDependencyFailure(std::shared_ptr<DB::LoadJob> const&, std::shared_ptr<DB::LoadJob> const&, std::exception_ptr&) @ 0x00000000135e7a76
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0d3bed
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0d3bad
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0d3b6d
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0d3b15
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013603b4e
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f708d
13. /build/src/Common/AsyncLoader.cpp:627: DB::AsyncLoader::finish(std::shared_ptr<DB::LoadJob> const&, DB::LoadStatus, std::exception_ptr, std::unique_lock<std::mutex>&) @ 0x00000000135eea1f
14. /build/src/Common/AsyncLoader.cpp:898: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f321e
15. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
22. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
23. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
25. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
26. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
27. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:16:52.410548 [ 16077 ] {} <Warning> Context: Server was built in debug mode. It will work slowly.
2024.11.05 20:16:52.445663 [ 16077 ] {} <Warning> Context: Server logging level is set to 'test' and performance is degraded. This cannot be used in production.
2024.11.05 20:16:52.447668 [ 16077 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
2024.11.05 20:16:52.450680 [ 16077 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.11.05 20:18:38.719069 [ 16077 ] {} <Warning> CloudPlacementInfo: Placement info has not been loaded
2024.11.05 20:18:39.148066 [ 16077 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.11.05 20:18:39.157132 [ 16077 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.11.05 20:18:39.157231 [ 16077 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.11.05 20:18:39.209295 [ 16810 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.11.05 20:18:44.315317 [ 16077 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.11.05 20:18:45.186698 [ 16077 ] {} <Warning> DatabaseReplicated (test_k6cbek3q): Database test_k6cbek3q exists locally, but marked dropped in ZooKeeper (/test/db/test_8/replicas/s1|r1). Will not try to start it up
2024.11.05 20:18:45.214868 [ 16077 ] {} <Warning> TablesLoader: Tables test_15.discounts_dict depend on default.discounts, but seems like that does not exist. Will ignore it and try to load existing tables
2024.11.05 20:18:46.452218 [ 16920 ] {} <Error> void DB::AsyncLoader::worker(Pool &): Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 20:18:53.753495 [ 16813 ] {} <Error> RaftInstance: failed to accept a rpc connection due to error 125, Operation canceled
2024.11.05 20:18:54.801837 [ 16077 ] {} <Error> Application: Code: 696. DB::Exception: Load job 'startup table test_15.wv' -> Code: 695. DB::Exception: Load job 'load table test_15.wv' failed: Code: 1. DB::Exception: Experimental WINDOW VIEW feature is not supported with new infrastructure for query analysis (the setting 'allow_experimental_analyzer'): Cannot attach table `test_15`.`wv` from metadata file /var/lib/clickhouse/store/c79/c79bb5f8-0428-41a4-8fc6-c36dbb1ba9d3/wv.sql from query ATTACH WINDOW VIEW test_15.wv UUID '9a989f3a-72b1-4dc5-925b-426f52db8230' TO test_15.dst (`count` UInt64, `w_end` DateTime('US/Samoa')) WATERMARK STRICTLY_ASCENDING AS SELECT count(a) AS count, hopEnd(wid) AS w_end FROM test_15.mt GROUP BY hop(timestamp, toIntervalSecond('2'), toIntervalSecond('3'), 'US/Samoa') AS wid. (UNSUPPORTED_METHOD), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000a8f56b7
6. /build/src/Storages/WindowView/StorageWindowView.cpp:1181: DB::StorageWindowView::StorageWindowView(DB::StorageID const&, std::shared_ptr<DB::Context const>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel) @ 0x000000001d17140d
7. /build/contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageWindowView* std::construct_at[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, DB::StorageWindowView*>(DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ade3
8. /build/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:302: void std::allocator_traits<std::allocator<DB::StorageWindowView>>::construct[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void, void>(std::allocator<DB::StorageWindowView>&, DB::StorageWindowView*, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18ab61
9. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: std::__shared_ptr_emplace<DB::StorageWindowView, std::allocator<DB::StorageWindowView>>::__shared_ptr_emplace[abi:v15000]<DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&>(std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a82d
10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageWindowView> std::allocate_shared[abi:v15000]<DB::StorageWindowView, std::allocator<DB::StorageWindowView>, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(std::allocator<DB::StorageWindowView> const&, DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a67a
11. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageWindowView> std::make_shared[abi:v15000]<DB::StorageWindowView, DB::StorageID const&, std::shared_ptr<DB::Context>, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&, void>(DB::StorageID const&, std::shared_ptr<DB::Context>&&, DB::ASTCreateQuery const&, DB::ColumnsDescription const&, DB::LoadingStrictnessLevel const&) @ 0x000000001d18a606
12. /build/src/Storages/WindowView/StorageWindowView.cpp:1693: DB::registerStorageWindowView(DB::StorageFactory&)::$_0::operator()(DB::StorageFactory::Arguments const&) const @ 0x000000001d17dfa6
13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001d17de1d
14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:470: ? @ 0x000000001d17ddbd
15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001d17dd7d
16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001d17dd45
17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001c3711de
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x000000001c36fafd
19. /build/src/Storages/StorageFactory.cpp:224: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x000000001c36e41d
20. /build/src/Databases/DatabaseOnDisk.cpp:120: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, DB::LoadingStrictnessLevel) @ 0x000000001a0b0f52
21. /build/src/Databases/DatabaseOrdinary.cpp:248: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x000000001a0d9c7b
22. /build/src/Databases/DatabaseOrdinary.cpp:283: DB::DatabaseOrdinary::loadTableFromMetadataAsync(DB::AsyncLoader&, std::unordered_set<std::shared_ptr<DB::LoadJob>, std::hash<std::shared_ptr<DB::LoadJob>>, std::equal_to<std::shared_ptr<DB::LoadJob>>, std::allocator<std::shared_ptr<DB::LoadJob>>>, std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel)::$_0::operator()(DB::AsyncLoader&, std::shared_ptr<DB::LoadJob> const&) const @ 0x000000001a0e1463
23. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0e13e5
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0e13a5
25. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0e1365
26. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0e1330
27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000135ffc46
28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f62c5
29. /build/src/Common/AsyncLoader.cpp:169: DB::LoadJob::execute(DB::AsyncLoader&, unsigned long, std::shared_ptr<DB::LoadJob> const&) @ 0x00000000135e810b
30. /build/src/Common/AsyncLoader.cpp:937: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f4767
31. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
 (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_FAILED) (version 24.3.12.76.altinitystable (altinity build)). (ASYNC_LOAD_CANCELED), Stack trace (when copying this message, always include the lines below):

0. /build/contrib/llvm-project/libcxx/include/exception:141: std::exception::capture() @ 0x000000000a8f9b82
1. /build/contrib/llvm-project/libcxx/include/exception:116: std::exception::exception[abi:v15000]() @ 0x000000000a8f9b4d
2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000022a0b3e0
3. /build/src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000132bbd2e
4. /build/src/Common/Exception.h:90: DB::Exception::Exception(String&&, int, bool) @ 0x000000000a8f072a
5. /build/src/Common/Exception.h:109: DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x000000000b4feccc
6. /build/src/Common/AsyncLoader.cpp:73: DB::cancelOnDependencyFailure(std::shared_ptr<DB::LoadJob> const&, std::shared_ptr<DB::LoadJob> const&, std::exception_ptr&) @ 0x00000000135e7a76
7. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x000000001a0d3bed
8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x000000001a0d3bad
9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x000000001a0d3b6d
10. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001a0d3b15
11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000013603b4e
12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x00000000135f708d
13. /build/src/Common/AsyncLoader.cpp:627: DB::AsyncLoader::finish(std::shared_ptr<DB::LoadJob> const&, DB::LoadStatus, std::exception_ptr, std::unique_lock<std::mutex>&) @ 0x00000000135eea1f
14. /build/src/Common/AsyncLoader.cpp:898: DB::AsyncLoader::worker(DB::AsyncLoader::Pool&) @ 0x00000000135f321e
15. /build/src/Common/AsyncLoader.cpp:872: DB::AsyncLoader::spawn(DB::AsyncLoader::Pool&, std::unique_lock<std::mutex>&)::$_1::operator()() const @ 0x00000000135f5adc
16. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000135f5ab5
17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000135f5a95
18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000135f5a75
19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000135f5a3d
20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x0000000011b449b6
21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: ? @ 0x0000000011b445f5
22. /build/src/Common/ThreadPool.cpp:526: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x00000000133d9882
23. /build/src/Common/ThreadPool.cpp:253: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()::operator()() const @ 0x00000000133e36c4
24. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3695
25. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x00000000133e3679
26. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x00000000133e351d
27. /build/src/Common/ThreadPool.h:225: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x00000000133e33e5
28. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000133e3355
29. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: ? @ 0x00000000133e3335
30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ? @ 0x00000000133e3315
31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x00000000133e32e0
 (version 24.3.12.76.altinitystable (altinity build))